climateprediction.net home page
Task 22257056

Task 22257056

Name oifs_43r3_ps_0921_1986050100_123_955_12172565_0
Workunit 12172565
Created 21 Dec 2022, 9:53:42 UTC
Sent 21 Dec 2022, 9:57:23 UTC
Report deadline 20 Jan 2023, 9:57:23 UTC
Received 1 Mar 2024, 16:29:30 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1514362
Run time 17 hours 31 min 26 sec
CPU time 16 hours 33 min 46 sec
Validate state Invalid
Credit 16,211.61
Device peak FLOPS 4.84 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,367.05 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,684.59 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
 to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002268
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_92.zip
Uploading the intermediate file: upload_file_92.zip
  02:15:53 STEP 2281 H=2281:00 +CPU= 16.274
Uploading trickle at timestep: 8208000
  02:16:10 STEP 2282 H=2282:00 +CPU= 16.643
  02:16:36 STEP 2283 H=2283:00 +CPU= 25.994
  02:16:53 STEP 2284 H=2284:00 +CPU= 16.607
  02:17:09 STEP 2285 H=2285:00 +CPU= 16.157
  02:17:35 STEP 2286 H=2286:00 +CPU= 26.290
  02:17:51 STEP 2287 H=2287:00 +CPU= 16.146
  02:18:08 STEP 2288 H=2288:00 +CPU= 16.634
  02:18:34 STEP 2289 H=2289:00 +CPU= 25.765
  02:18:50 STEP 2290 H=2290:00 +CPU= 16.617
  02:19:06 STEP 2291 H=2291:00 +CPU= 16.115
  02:19:35 STEP 2292 H=2292:00 +CPU= 28.437
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002292
  02:19:51 STEP 2293 H=2293:00 +CPU= 16.144
  02:20:08 STEP 2294 H=2294:00 +CPU= 16.603
  02:20:33 STEP 2295 H=2295:00 +CPU= 25.711
  02:20:50 STEP 2296 H=2296:00 +CPU= 16.600
  02:21:06 STEP 2297 H=2297:00 +CPU= 16.114
  02:21:32 STEP 2298 H=2298:00 +CPU= 26.262
  02:21:48 STEP 2299 H=2299:00 +CPU= 16.147
  02:22:05 STEP 2300 H=2300:00 +CPU= 16.575
  02:22:31 STEP 2301 H=2301:00 +CPU= 25.742
  02:22:47 STEP 2302 H=2302:00 +CPU= 16.601
  02:23:04 STEP 2303 H=2303:00 +CPU= 16.626
  02:23:35 STEP 2304 H=2304:00 +CPU= 30.411
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002280
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002280
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002280
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002292
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002292
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002292
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_93.zip
Uploading the intermediate file: upload_file_93.zip
  02:23:51 STEP 2305 H=2305:00 +CPU= 16.176
  02:24:07 STEP 2306 H=2306:00 +CPU= 16.693
  02:24:33 STEP 2307 H=2307:00 +CPU= 25.900
  02:24:50 STEP 2308 H=2308:00 +CPU= 16.593
  02:25:06 STEP 2309 H=2309:00 +CPU= 16.171
  02:25:32 STEP 2310 H=2310:00 +CPU= 26.240
  02:25:49 STEP 2311 H=2311:00 +CPU= 16.132
  02:26:05 STEP 2312 H=2312:00 +CPU= 16.620
  02:26:31 STEP 2313 H=2313:00 +CPU= 25.740
  02:26:48 STEP 2314 H=2314:00 +CPU= 16.603
  02:27:04 STEP 2315 H=2315:00 +CPU= 16.117
  02:27:32 STEP 2316 H=2316:00 +CPU= 28.460
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002316
  02:27:48 STEP 2317 H=2317:00 +CPU= 16.136
  02:28:05 STEP 2318 H=2318:00 +CPU= 16.605
  02:28:31 STEP 2319 H=2319:00 +CPU= 25.727
  02:28:47 STEP 2320 H=2320:00 +CPU= 16.569
  02:29:03 STEP 2321 H=2321:00 +CPU= 16.084
  02:29:30 STEP 2322 H=2322:00 +CPU= 26.197
  02:29:46 STEP 2323 H=2323:00 +CPU= 16.091
  02:30:02 STEP 2324 H=2324:00 +CPU= 16.557
  02:30:28 STEP 2325 H=2325:00 +CPU= 25.636
  02:30:44 STEP 2326 H=2326:00 +CPU= 16.547
  02:31:01 STEP 2327 H=2327:00 +CPU= 16.568
  02:31:31 STEP 2328 H=2328:00 +CPU= 30.308
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002304
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002304
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002304
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002316
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002316
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002316
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_94.zip
Uploading the intermediate file: upload_file_94.zip
  02:31:48 STEP 2329 H=2329:00 +CPU= 16.176
  02:32:04 STEP 2330 H=2330:00 +CPU= 16.548
  02:32:30 STEP 2331 H=2331:00 +CPU= 25.877
  02:32:47 STEP 2332 H=2332:00 +CPU= 16.558
  02:33:03 STEP 2333 H=2333:00 +CPU= 16.107
  02:33:29 STEP 2334 H=2334:00 +CPU= 26.253
  02:33:45 STEP 2335 H=2335:00 +CPU= 16.110
  02:34:02 STEP 2336 H=2336:00 +CPU= 16.585
  02:34:27 STEP 2337 H=2337:00 +CPU= 25.675
  02:34:44 STEP 2338 H=2338:00 +CPU= 16.608
  02:35:00 STEP 2339 H=2339:00 +CPU= 16.118
  02:35:29 STEP 2340 H=2340:00 +CPU= 28.438
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002340
  02:35:45 STEP 2341 H=2341:00 +CPU= 16.124
  02:36:01 STEP 2342 H=2342:00 +CPU= 16.612
  02:36:27 STEP 2343 H=2343:00 +CPU= 25.652
  02:36:44 STEP 2344 H=2344:00 +CPU= 16.573
  02:37:00 STEP 2345 H=2345:00 +CPU= 16.059
  02:37:26 STEP 2346 H=2346:00 +CPU= 26.207
  02:37:42 STEP 2347 H=2347:00 +CPU= 16.117
  02:37:59 STEP 2348 H=2348:00 +CPU= 16.535
  02:38:24 STEP 2349 H=2349:00 +CPU= 25.681
  02:38:41 STEP 2350 H=2350:00 +CPU= 16.580
  02:38:57 STEP 2351 H=2351:00 +CPU= 16.610
  02:39:28 STEP 2352 H=2352:00 +CPU= 30.347
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002328
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002328
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002328
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002340
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002340
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002340
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_95.zip
Uploading the intermediate file: upload_file_95.zip
  02:39:44 STEP 2353 H=2353:00 +CPU= 16.180
  02:40:01 STEP 2354 H=2354:00 +CPU= 16.566
  02:40:27 STEP 2355 H=2355:00 +CPU= 25.899
  02:40:43 STEP 2356 H=2356:00 +CPU= 16.556
  02:40:59 STEP 2357 H=2357:00 +CPU= 16.109
  02:41:26 STEP 2358 H=2358:00 +CPU= 26.275
  02:41:42 STEP 2359 H=2359:00 +CPU= 16.145
  02:41:58 STEP 2360 H=2360:00 +CPU= 16.613
  02:42:24 STEP 2361 H=2361:00 +CPU= 25.722
  02:42:41 STEP 2362 H=2362:00 +CPU= 16.598
  02:42:57 STEP 2363 H=2363:00 +CPU= 16.108
  02:43:25 STEP 2364 H=2364:00 +CPU= 28.427
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002364
  02:43:41 STEP 2365 H=2365:00 +CPU= 16.166
  02:43:58 STEP 2366 H=2366:00 +CPU= 16.617
  02:44:24 STEP 2367 H=2367:00 +CPU= 25.678
  02:44:40 STEP 2368 H=2368:00 +CPU= 16.573
  02:44:56 STEP 2369 H=2369:00 +CPU= 16.075
  02:45:23 STEP 2370 H=2370:00 +CPU= 26.190
  02:45:39 STEP 2371 H=2371:00 +CPU= 16.095
  02:45:55 STEP 2372 H=2372:00 +CPU= 16.547
  02:46:21 STEP 2373 H=2373:00 +CPU= 25.677
  02:46:38 STEP 2374 H=2374:00 +CPU= 16.584
  02:46:54 STEP 2375 H=2375:00 +CPU= 16.615
  02:47:25 STEP 2376 H=2376:00 +CPU= 30.409
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002352
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002352
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002352
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002364
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002364
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002364
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_96.zip
Uploading the intermediate file: upload_file_96.zip
  02:47:41 STEP 2377 H=2377:00 +CPU= 16.178
  02:47:58 STEP 2378 H=2378:00 +CPU= 16.623
  02:48:23 STEP 2379 H=2379:00 +CPU= 25.826
  02:48:40 STEP 2380 H=2380:00 +CPU= 16.660
  02:48:56 STEP 2381 H=2381:00 +CPU= 16.148
  02:49:23 STEP 2382 H=2382:00 +CPU= 26.314
  02:49:39 STEP 2383 H=2383:00 +CPU= 16.135
  02:49:55 STEP 2384 H=2384:00 +CPU= 16.602
  02:50:21 STEP 2385 H=2385:00 +CPU= 25.710
  02:50:38 STEP 2386 H=2386:00 +CPU= 16.641
  02:50:54 STEP 2387 H=2387:00 +CPU= 16.145
  02:51:22 STEP 2388 H=2388:00 +CPU= 28.512
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002388
  02:51:39 STEP 2389 H=2389:00 +CPU= 16.121
  02:51:55 STEP 2390 H=2390:00 +CPU= 16.604
  02:52:21 STEP 2391 H=2391:00 +CPU= 25.662
  02:52:37 STEP 2392 H=2392:00 +CPU= 16.605
  02:52:54 STEP 2393 H=2393:00 +CPU= 16.115
  02:53:20 STEP 2394 H=2394:00 +CPU= 26.276
  02:53:36 STEP 2395 H=2395:00 +CPU= 16.108
  02:53:53 STEP 2396 H=2396:00 +CPU= 16.600
  02:54:18 STEP 2397 H=2397:00 +CPU= 25.738
  02:54:35 STEP 2398 H=2398:00 +CPU= 16.602
  02:54:52 STEP 2399 H=2399:00 +CPU= 16.619
  02:55:22 STEP 2400 H=2400:00 +CPU= 30.443
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002376
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002376
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002376
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002388
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002388
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002388
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_97.zip
Uploading the intermediate file: upload_file_97.zip
  02:55:38 STEP 2401 H=2401:00 +CPU= 16.135
  02:55:55 STEP 2402 H=2402:00 +CPU= 16.631
  02:56:21 STEP 2403 H=2403:00 +CPU= 25.932
  02:56:37 STEP 2404 H=2404:00 +CPU= 16.591
  02:56:54 STEP 2405 H=2405:00 +CPU= 16.115
  02:57:20 STEP 2406 H=2406:00 +CPU= 26.292
  02:57:36 STEP 2407 H=2407:00 +CPU= 16.133
  02:57:53 STEP 2408 H=2408:00 +CPU= 16.647
  02:58:18 STEP 2409 H=2409:00 +CPU= 25.746
  02:58:35 STEP 2410 H=2410:00 +CPU= 16.642
  02:58:51 STEP 2411 H=2411:00 +CPU= 16.135
  02:59:20 STEP 2412 H=2412:00 +CPU= 28.504
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002412
  02:59:36 STEP 2413 H=2413:00 +CPU= 16.135
  02:59:53 STEP 2414 H=2414:00 +CPU= 16.615
  03:00:18 STEP 2415 H=2415:00 +CPU= 25.730
  03:00:35 STEP 2416 H=2416:00 +CPU= 16.598
  03:00:51 STEP 2417 H=2417:00 +CPU= 16.076
  03:01:17 STEP 2418 H=2418:00 +CPU= 26.242
  03:01:33 STEP 2419 H=2419:00 +CPU= 16.129
  03:01:50 STEP 2420 H=2420:00 +CPU= 16.603
  03:02:16 STEP 2421 H=2421:00 +CPU= 25.677
  03:02:32 STEP 2422 H=2422:00 +CPU= 16.598
  03:02:49 STEP 2423 H=2423:00 +CPU= 16.591
  03:03:19 STEP 2424 H=2424:00 +CPU= 30.399
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002400
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002400
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002400
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002412
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002412
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002412
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_98.zip
Uploading the intermediate file: upload_file_98.zip
  03:03:36 STEP 2425 H=2425:00 +CPU= 16.167
  03:03:52 STEP 2426 H=2426:00 +CPU= 16.652
  03:04:18 STEP 2427 H=2427:00 +CPU= 25.849
  03:04:35 STEP 2428 H=2428:00 +CPU= 16.603
  03:04:51 STEP 2429 H=2429:00 +CPU= 16.116
  03:05:17 STEP 2430 H=2430:00 +CPU= 26.300
  03:05:33 STEP 2431 H=2431:00 +CPU= 16.160
  03:05:50 STEP 2432 H=2432:00 +CPU= 16.667
  03:06:16 STEP 2433 H=2433:00 +CPU= 25.752
  03:06:32 STEP 2434 H=2434:00 +CPU= 16.625
  03:06:49 STEP 2435 H=2435:00 +CPU= 16.160
  03:07:17 STEP 2436 H=2436:00 +CPU= 28.538
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002436
  03:07:33 STEP 2437 H=2437:00 +CPU= 16.170
  03:07:50 STEP 2438 H=2438:00 +CPU= 16.611
  03:08:16 STEP 2439 H=2439:00 +CPU= 25.715
  03:08:32 STEP 2440 H=2440:00 +CPU= 16.611
  03:08:49 STEP 2441 H=2441:00 +CPU= 16.141
  03:09:15 STEP 2442 H=2442:00 +CPU= 26.249
  03:09:31 STEP 2443 H=2443:00 +CPU= 16.117
  03:09:48 STEP 2444 H=2444:00 +CPU= 16.566
  03:10:13 STEP 2445 H=2445:00 +CPU= 25.685
  03:10:30 STEP 2446 H=2446:00 +CPU= 16.586
  03:10:46 STEP 2447 H=2447:00 +CPU= 16.605
  03:11:17 STEP 2448 H=2448:00 +CPU= 30.421
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002424
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002424
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002424
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002436
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002436
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002436
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_99.zip
Uploading the intermediate file: upload_file_99.zip
  03:11:33 STEP 2449 H=2449:00 +CPU= 16.143
  03:11:50 STEP 2450 H=2450:00 +CPU= 16.597
  03:12:16 STEP 2451 H=2451:00 +CPU= 25.906
  03:12:32 STEP 2452 H=2452:00 +CPU= 16.580
  03:12:48 STEP 2453 H=2453:00 +CPU= 16.096
  03:13:15 STEP 2454 H=2454:00 +CPU= 26.246
  03:13:31 STEP 2455 H=2455:00 +CPU= 16.120
  03:13:47 STEP 2456 H=2456:00 +CPU= 16.627
  03:14:13 STEP 2457 H=2457:00 +CPU= 25.776
  03:14:30 STEP 2458 H=2458:00 +CPU= 16.614
  03:14:46 STEP 2459 H=2459:00 +CPU= 16.149
  03:15:14 STEP 2460 H=2460:00 +CPU= 28.498
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002460
  03:15:31 STEP 2461 H=2461:00 +CPU= 16.122
  03:15:47 STEP 2462 H=2462:00 +CPU= 16.657
  03:16:13 STEP 2463 H=2463:00 +CPU= 25.746
  03:16:30 STEP 2464 H=2464:00 +CPU= 16.620
  03:16:46 STEP 2465 H=2465:00 +CPU= 16.112
  03:17:12 STEP 2466 H=2466:00 +CPU= 26.196
  03:17:28 STEP 2467 H=2467:00 +CPU= 16.134
  03:17:45 STEP 2468 H=2468:00 +CPU= 16.571
  03:18:10 STEP 2469 H=2469:00 +CPU= 25.726
  03:18:27 STEP 2470 H=2470:00 +CPU= 16.578
  03:18:44 STEP 2471 H=2471:00 +CPU= 16.590
  03:19:14 STEP 2472 H=2472:00 +CPU= 30.330
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002448
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002448
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002448
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002460
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002460
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002460
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_100.zip
Uploading the intermediate file: upload_file_100.zip
  03:19:30 STEP 2473 H=2473:00 +CPU= 16.177
  03:19:47 STEP 2474 H=2474:00 +CPU= 16.546
  03:20:13 STEP 2475 H=2475:00 +CPU= 25.877
  03:20:29 STEP 2476 H=2476:00 +CPU= 16.569
  03:20:45 STEP 2477 H=2477:00 +CPU= 16.108
  03:21:12 STEP 2478 H=2478:00 +CPU= 26.276
  03:21:28 STEP 2479 H=2479:00 +CPU= 16.113
  03:21:44 STEP 2480 H=2480:00 +CPU= 16.586
  03:22:10 STEP 2481 H=2481:00 +CPU= 25.717
  03:22:27 STEP 2482 H=2482:00 +CPU= 16.621
  03:22:43 STEP 2483 H=2483:00 +CPU= 16.138
  03:23:11 STEP 2484 H=2484:00 +CPU= 28.497
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002484
  03:23:28 STEP 2485 H=2485:00 +CPU= 16.168
  03:23:44 STEP 2486 H=2486:00 +CPU= 16.635
  03:24:10 STEP 2487 H=2487:00 +CPU= 25.713
  03:24:27 STEP 2488 H=2488:00 +CPU= 16.623
  03:24:43 STEP 2489 H=2489:00 +CPU= 16.124
  03:25:09 STEP 2490 H=2490:00 +CPU= 26.183
  03:25:25 STEP 2491 H=2491:00 +CPU= 16.098
  03:25:42 STEP 2492 H=2492:00 +CPU= 16.559
  03:26:07 STEP 2493 H=2493:00 +CPU= 25.699
  03:26:24 STEP 2494 H=2494:00 +CPU= 16.514
  03:26:41 STEP 2495 H=2495:00 +CPU= 16.568
  03:27:11 STEP 2496 H=2496:00 +CPU= 30.308
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002472
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002472
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002472
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002484
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002484
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002484
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_101.zip
Uploading the intermediate file: upload_file_101.zip
  03:27:27 STEP 2497 H=2497:00 +CPU= 16.223
  03:27:44 STEP 2498 H=2498:00 +CPU= 16.602
  03:28:10 STEP 2499 H=2499:00 +CPU= 25.790
  03:28:26 STEP 2500 H=2500:00 +CPU= 16.583
  03:28:42 STEP 2501 H=2501:00 +CPU= 16.097
  03:29:08 STEP 2502 H=2502:00 +CPU= 26.201
  03:29:25 STEP 2503 H=2503:00 +CPU= 16.105
  03:29:41 STEP 2504 H=2504:00 +CPU= 16.605
  03:30:07 STEP 2505 H=2505:00 +CPU= 25.661
  03:30:23 STEP 2506 H=2506:00 +CPU= 16.598
  03:30:40 STEP 2507 H=2507:00 +CPU= 16.104
  03:31:08 STEP 2508 H=2508:00 +CPU= 28.460
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002508
  03:31:24 STEP 2509 H=2509:00 +CPU= 16.137
  03:31:41 STEP 2510 H=2510:00 +CPU= 16.564
  03:32:07 STEP 2511 H=2511:00 +CPU= 25.688
  03:32:23 STEP 2512 H=2512:00 +CPU= 16.557
  03:32:39 STEP 2513 H=2513:00 +CPU= 16.087
  03:33:05 STEP 2514 H=2514:00 +CPU= 26.114
  03:33:21 STEP 2515 H=2515:00 +CPU= 16.085
  03:33:38 STEP 2516 H=2516:00 +CPU= 16.528
  03:34:04 STEP 2517 H=2517:00 +CPU= 25.615
  03:34:20 STEP 2518 H=2518:00 +CPU= 16.549
  03:34:37 STEP 2519 H=2519:00 +CPU= 16.561
  03:35:07 STEP 2520 H=2520:00 +CPU= 30.367
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002496
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002496
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002496
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002508
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002508
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002508
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_102.zip
Uploading the intermediate file: upload_file_102.zip
  03:35:23 STEP 2521 H=2521:00 +CPU= 16.117
Uploading trickle at timestep: 9072000
  03:35:40 STEP 2522 H=2522:00 +CPU= 16.497
  03:36:06 STEP 2523 H=2523:00 +CPU= 25.865
  03:36:22 STEP 2524 H=2524:00 +CPU= 16.576
  03:36:38 STEP 2525 H=2525:00 +CPU= 16.097
  03:37:05 STEP 2526 H=2526:00 +CPU= 26.236
  03:37:21 STEP 2527 H=2527:00 +CPU= 16.105
  03:37:37 STEP 2528 H=2528:00 +CPU= 16.569
  03:38:03 STEP 2529 H=2529:00 +CPU= 25.658
  03:38:20 STEP 2530 H=2530:00 +CPU= 16.575
  03:38:36 STEP 2531 H=2531:00 +CPU= 16.098
  03:39:04 STEP 2532 H=2532:00 +CPU= 28.411
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002532
  03:39:20 STEP 2533 H=2533:00 +CPU= 16.104
  03:39:37 STEP 2534 H=2534:00 +CPU= 16.522
  03:40:02 STEP 2535 H=2535:00 +CPU= 25.571
  03:40:19 STEP 2536 H=2536:00 +CPU= 16.550
  03:40:35 STEP 2537 H=2537:00 +CPU= 16.059
  03:41:01 STEP 2538 H=2538:00 +CPU= 26.156
  03:41:17 STEP 2539 H=2539:00 +CPU= 16.070
  03:41:34 STEP 2540 H=2540:00 +CPU= 16.542
  03:41:59 STEP 2541 H=2541:00 +CPU= 25.578
  03:42:16 STEP 2542 H=2542:00 +CPU= 16.516
  03:42:33 STEP 2543 H=2543:00 +CPU= 16.562
  03:43:03 STEP 2544 H=2544:00 +CPU= 30.306
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002520
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002520
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002520
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002532
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002532
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002532
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_103.zip
Uploading the intermediate file: upload_file_103.zip
  03:43:19 STEP 2545 H=2545:00 +CPU= 16.191
  03:43:36 STEP 2546 H=2546:00 +CPU= 16.595
  03:44:02 STEP 2547 H=2547:00 +CPU= 25.752
  03:44:18 STEP 2548 H=2548:00 +CPU= 16.573
  03:44:34 STEP 2549 H=2549:00 +CPU= 16.111
  03:45:00 STEP 2550 H=2550:00 +CPU= 26.202
  03:45:17 STEP 2551 H=2551:00 +CPU= 16.125
  03:45:33 STEP 2552 H=2552:00 +CPU= 16.608
  03:45:59 STEP 2553 H=2553:00 +CPU= 25.683
  03:46:16 STEP 2554 H=2554:00 +CPU= 16.645
  03:46:32 STEP 2555 H=2555:00 +CPU= 16.141
  03:47:00 STEP 2556 H=2556:00 +CPU= 28.420
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002556
  03:47:16 STEP 2557 H=2557:00 +CPU= 16.115
  03:47:33 STEP 2558 H=2558:00 +CPU= 16.586
  03:47:59 STEP 2559 H=2559:00 +CPU= 25.600
  03:48:15 STEP 2560 H=2560:00 +CPU= 16.589
  03:48:31 STEP 2561 H=2561:00 +CPU= 16.104
  03:48:57 STEP 2562 H=2562:00 +CPU= 26.172
  03:49:14 STEP 2563 H=2563:00 +CPU= 16.081
  03:49:30 STEP 2564 H=2564:00 +CPU= 16.564
  03:49:56 STEP 2565 H=2565:00 +CPU= 25.612
  03:50:12 STEP 2566 H=2566:00 +CPU= 16.566
  03:50:29 STEP 2567 H=2567:00 +CPU= 16.586
  03:50:59 STEP 2568 H=2568:00 +CPU= 30.289
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002544
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002544
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002544
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002556
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002556
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002556
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_104.zip
Uploading the intermediate file: upload_file_104.zip
  03:51:15 STEP 2569 H=2569:00 +CPU= 16.155
  03:51:32 STEP 2570 H=2570:00 +CPU= 16.557
  03:51:58 STEP 2571 H=2571:00 +CPU= 25.837
  03:52:14 STEP 2572 H=2572:00 +CPU= 16.556
  03:52:31 STEP 2573 H=2573:00 +CPU= 16.086
  03:52:57 STEP 2574 H=2574:00 +CPU= 26.224
  03:53:13 STEP 2575 H=2575:00 +CPU= 16.111
  03:53:30 STEP 2576 H=2576:00 +CPU= 16.605
  03:53:55 STEP 2577 H=2577:00 +CPU= 25.645
  03:54:12 STEP 2578 H=2578:00 +CPU= 16.608
  03:54:28 STEP 2579 H=2579:00 +CPU= 16.118
  03:54:56 STEP 2580 H=2580:00 +CPU= 28.469
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002580
  03:55:13 STEP 2581 H=2581:00 +CPU= 16.125
  03:55:29 STEP 2582 H=2582:00 +CPU= 16.576
  03:55:55 STEP 2583 H=2583:00 +CPU= 25.659
  03:56:11 STEP 2584 H=2584:00 +CPU= 16.595
  03:56:28 STEP 2585 H=2585:00 +CPU= 16.100
  03:56:54 STEP 2586 H=2586:00 +CPU= 26.169
  03:57:10 STEP 2587 H=2587:00 +CPU= 16.088
  03:57:26 STEP 2588 H=2588:00 +CPU= 16.575
  03:57:52 STEP 2589 H=2589:00 +CPU= 25.624
  03:58:09 STEP 2590 H=2590:00 +CPU= 16.563
  03:58:25 STEP 2591 H=2591:00 +CPU= 16.592
  03:58:56 STEP 2592 H=2592:00 +CPU= 30.285
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002568
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002568
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002568
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002580
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002580
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002580
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_105.zip
Uploading the intermediate file: upload_file_105.zip
  03:59:12 STEP 2593 H=2593:00 +CPU= 16.179
  03:59:28 STEP 2594 H=2594:00 +CPU= 16.638
  03:59:54 STEP 2595 H=2595:00 +CPU= 25.797
  04:00:11 STEP 2596 H=2596:00 +CPU= 16.570
  04:00:27 STEP 2597 H=2597:00 +CPU= 16.106
  04:00:53 STEP 2598 H=2598:00 +CPU= 26.236
  04:01:09 STEP 2599 H=2599:00 +CPU= 16.144
  04:01:26 STEP 2600 H=2600:00 +CPU= 16.600
  04:01:52 STEP 2601 H=2601:00 +CPU= 25.673
  04:02:08 STEP 2602 H=2602:00 +CPU= 16.613
  04:02:24 STEP 2603 H=2603:00 +CPU= 16.146
  04:02:53 STEP 2604 H=2604:00 +CPU= 28.475
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002604
  04:03:09 STEP 2605 H=2605:00 +CPU= 16.107
  04:03:26 STEP 2606 H=2606:00 +CPU= 16.608
  04:03:51 STEP 2607 H=2607:00 +CPU= 25.689
  04:04:08 STEP 2608 H=2608:00 +CPU= 16.604
  04:04:24 STEP 2609 H=2609:00 +CPU= 16.077
  04:04:50 STEP 2610 H=2610:00 +CPU= 26.185
  04:05:06 STEP 2611 H=2611:00 +CPU= 16.097
  04:05:23 STEP 2612 H=2612:00 +CPU= 16.582
  04:05:49 STEP 2613 H=2613:00 +CPU= 25.678
  04:06:05 STEP 2614 H=2614:00 +CPU= 16.575
  04:06:22 STEP 2615 H=2615:00 +CPU= 16.583
  04:06:52 STEP 2616 H=2616:00 +CPU= 30.299
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002592
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002592
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002592
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002604
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002604
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002604
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_106.zip
Uploading the intermediate file: upload_file_106.zip
  04:07:08 STEP 2617 H=2617:00 +CPU= 16.155
  04:07:25 STEP 2618 H=2618:00 +CPU= 16.577
  04:07:51 STEP 2619 H=2619:00 +CPU= 25.895
  04:08:07 STEP 2620 H=2620:00 +CPU= 16.581
  04:08:24 STEP 2621 H=2621:00 +CPU= 16.109
  04:08:50 STEP 2622 H=2622:00 +CPU= 26.250
  04:09:06 STEP 2623 H=2623:00 +CPU= 16.153
  04:09:23 STEP 2624 H=2624:00 +CPU= 16.590
  04:09:48 STEP 2625 H=2625:00 +CPU= 25.689
  04:10:05 STEP 2626 H=2626:00 +CPU= 16.630
  04:10:21 STEP 2627 H=2627:00 +CPU= 16.137
  04:10:50 STEP 2628 H=2628:00 +CPU= 28.463
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002628
  04:11:06 STEP 2629 H=2629:00 +CPU= 16.157
  04:11:22 STEP 2630 H=2630:00 +CPU= 16.638
  04:11:48 STEP 2631 H=2631:00 +CPU= 25.693
  04:12:05 STEP 2632 H=2632:00 +CPU= 16.589
  04:12:21 STEP 2633 H=2633:00 +CPU= 16.104
  04:12:47 STEP 2634 H=2634:00 +CPU= 26.194
  04:13:03 STEP 2635 H=2635:00 +CPU= 16.086
  04:13:20 STEP 2636 H=2636:00 +CPU= 16.579
  04:13:45 STEP 2637 H=2637:00 +CPU= 25.632
  04:14:02 STEP 2638 H=2638:00 +CPU= 16.585
  04:14:19 STEP 2639 H=2639:00 +CPU= 16.579
  04:14:49 STEP 2640 H=2640:00 +CPU= 30.307
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002640
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002640
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002640
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002616
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002616
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002616
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002628
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002628
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002628
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_107.zip
Uploading the intermediate file: upload_file_107.zip
  04:15:05 STEP 2641 H=2641:00 +CPU= 16.169
  04:15:22 STEP 2642 H=2642:00 +CPU= 16.668
  04:15:48 STEP 2643 H=2643:00 +CPU= 25.743
  04:16:04 STEP 2644 H=2644:00 +CPU= 16.593
  04:16:20 STEP 2645 H=2645:00 +CPU= 16.104
  04:16:47 STEP 2646 H=2646:00 +CPU= 26.286
  04:17:03 STEP 2647 H=2647:00 +CPU= 16.109
  04:17:19 STEP 2648 H=2648:00 +CPU= 16.615
  04:17:45 STEP 2649 H=2649:00 +CPU= 25.714
  04:18:02 STEP 2650 H=2650:00 +CPU= 16.632
  04:18:18 STEP 2651 H=2651:00 +CPU= 16.132
  04:18:46 STEP 2652 H=2652:00 +CPU= 28.515
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002652
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002652
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002652
  04:19:03 STEP 2653 H=2653:00 +CPU= 16.118
  04:19:19 STEP 2654 H=2654:00 +CPU= 16.628
  04:19:45 STEP 2655 H=2655:00 +CPU= 25.699
  04:20:02 STEP 2656 H=2656:00 +CPU= 16.610
  04:20:18 STEP 2657 H=2657:00 +CPU= 16.083
  04:20:44 STEP 2658 H=2658:00 +CPU= 26.218
  04:21:00 STEP 2659 H=2659:00 +CPU= 16.105
  04:21:17 STEP 2660 H=2660:00 +CPU= 16.588
  04:21:42 STEP 2661 H=2661:00 +CPU= 25.668
  04:21:59 STEP 2662 H=2662:00 +CPU= 16.567
  04:22:16 STEP 2663 H=2663:00 +CPU= 16.594
  04:22:46 STEP 2664 H=2664:00 +CPU= 30.389
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002664
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002664
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002664
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002640
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002640
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002640
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002652
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002652
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002652
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_108.zip
Uploading the intermediate file: upload_file_108.zip
  04:23:02 STEP 2665 H=2665:00 +CPU= 16.179
  04:23:19 STEP 2666 H=2666:00 +CPU= 16.597
  04:23:45 STEP 2667 H=2667:00 +CPU= 25.928
  04:24:01 STEP 2668 H=2668:00 +CPU= 16.684
  04:24:18 STEP 2669 H=2669:00 +CPU= 16.153
  04:24:44 STEP 2670 H=2670:00 +CPU= 26.359
  04:25:00 STEP 2671 H=2671:00 +CPU= 16.154
  04:25:17 STEP 2672 H=2672:00 +CPU= 16.651
  04:25:43 STEP 2673 H=2673:00 +CPU= 25.773
  04:25:59 STEP 2674 H=2674:00 +CPU= 16.660
  04:26:15 STEP 2675 H=2675:00 +CPU= 16.151
  04:26:44 STEP 2676 H=2676:00 +CPU= 28.541
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002676
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002676
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002676
  04:27:00 STEP 2677 H=2677:00 +CPU= 16.154
  04:27:17 STEP 2678 H=2678:00 +CPU= 16.600
  04:27:43 STEP 2679 H=2679:00 +CPU= 25.731
  04:27:59 STEP 2680 H=2680:00 +CPU= 16.603
  04:28:15 STEP 2681 H=2681:00 +CPU= 16.189
  04:28:42 STEP 2682 H=2682:00 +CPU= 26.222
  04:28:58 STEP 2683 H=2683:00 +CPU= 16.106
  04:29:14 STEP 2684 H=2684:00 +CPU= 16.586
  04:29:40 STEP 2685 H=2685:00 +CPU= 25.679
  04:29:57 STEP 2686 H=2686:00 +CPU= 16.603
  04:30:13 STEP 2687 H=2687:00 +CPU= 16.563
  04:30:44 STEP 2688 H=2688:00 +CPU= 30.313
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002688
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002688
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002688
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002664
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002664
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002664
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002676
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002676
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002676
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_109.zip
Uploading the intermediate file: upload_file_109.zip
  04:31:00 STEP 2689 H=2689:00 +CPU= 16.173
  04:31:16 STEP 2690 H=2690:00 +CPU= 16.663
  04:31:42 STEP 2691 H=2691:00 +CPU= 25.847
  04:31:59 STEP 2692 H=2692:00 +CPU= 16.610
  04:32:15 STEP 2693 H=2693:00 +CPU= 16.112
  04:32:41 STEP 2694 H=2694:00 +CPU= 26.267
  04:32:57 STEP 2695 H=2695:00 +CPU= 16.141
  04:33:14 STEP 2696 H=2696:00 +CPU= 16.626
  04:33:40 STEP 2697 H=2697:00 +CPU= 25.734
  04:33:57 STEP 2698 H=2698:00 +CPU= 16.632
  04:34:13 STEP 2699 H=2699:00 +CPU= 16.159
  04:34:41 STEP 2700 H=2700:00 +CPU= 28.548
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002700
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002700
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002700
  04:34:57 STEP 2701 H=2701:00 +CPU= 16.137
  04:35:14 STEP 2702 H=2702:00 +CPU= 16.574
  04:35:40 STEP 2703 H=2703:00 +CPU= 25.718
  04:35:56 STEP 2704 H=2704:00 +CPU= 16.645
  04:36:13 STEP 2705 H=2705:00 +CPU= 16.127
  04:36:39 STEP 2706 H=2706:00 +CPU= 26.274
  04:36:55 STEP 2707 H=2707:00 +CPU= 16.119
  04:37:12 STEP 2708 H=2708:00 +CPU= 16.604
  04:37:37 STEP 2709 H=2709:00 +CPU= 25.690
  04:37:54 STEP 2710 H=2710:00 +CPU= 16.576
  04:38:11 STEP 2711 H=2711:00 +CPU= 16.609
  04:38:41 STEP 2712 H=2712:00 +CPU= 30.326
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002712
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002712
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002712
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002688
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002688
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002688
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002700
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002700
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002700
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_110.zip
Uploading the intermediate file: upload_file_110.zip
  04:38:57 STEP 2713 H=2713:00 +CPU= 16.153
  04:39:14 STEP 2714 H=2714:00 +CPU= 16.660
  04:39:40 STEP 2715 H=2715:00 +CPU= 25.914
  04:39:56 STEP 2716 H=2716:00 +CPU= 16.635
  04:40:12 STEP 2717 H=2717:00 +CPU= 16.121
  04:40:39 STEP 2718 H=2718:00 +CPU= 26.262
  04:40:55 STEP 2719 H=2719:00 +CPU= 16.159
  04:41:12 STEP 2720 H=2720:00 +CPU= 16.641
  04:41:37 STEP 2721 H=2721:00 +CPU= 25.697
  04:41:54 STEP 2722 H=2722:00 +CPU= 16.631
  04:42:10 STEP 2723 H=2723:00 +CPU= 16.126
  04:42:39 STEP 2724 H=2724:00 +CPU= 28.511
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002724
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002724
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002724
  04:42:55 STEP 2725 H=2725:00 +CPU= 16.141
  04:43:11 STEP 2726 H=2726:00 +CPU= 16.636
  04:43:37 STEP 2727 H=2727:00 +CPU= 25.701
  04:43:54 STEP 2728 H=2728:00 +CPU= 16.616
  04:44:10 STEP 2729 H=2729:00 +CPU= 16.140
  04:44:36 STEP 2730 H=2730:00 +CPU= 26.262
  04:44:52 STEP 2731 H=2731:00 +CPU= 16.104
  04:45:09 STEP 2732 H=2732:00 +CPU= 16.597
  04:45:35 STEP 2733 H=2733:00 +CPU= 25.717
  04:45:51 STEP 2734 H=2734:00 +CPU= 16.650
  04:46:08 STEP 2735 H=2735:00 +CPU= 16.616
  04:46:38 STEP 2736 H=2736:00 +CPU= 30.412
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002736
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002736
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002736
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002712
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002712
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002712
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002724
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002724
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002724
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_111.zip
Uploading the intermediate file: upload_file_111.zip
  04:46:55 STEP 2737 H=2737:00 +CPU= 16.139
  04:47:11 STEP 2738 H=2738:00 +CPU= 16.611
  04:47:37 STEP 2739 H=2739:00 +CPU= 25.937
  04:47:54 STEP 2740 H=2740:00 +CPU= 16.606
  04:48:10 STEP 2741 H=2741:00 +CPU= 16.122
  04:48:36 STEP 2742 H=2742:00 +CPU= 26.307
  04:48:52 STEP 2743 H=2743:00 +CPU= 16.140
  04:49:09 STEP 2744 H=2744:00 +CPU= 16.624
  04:49:35 STEP 2745 H=2745:00 +CPU= 25.730
  04:49:51 STEP 2746 H=2746:00 +CPU= 16.617
  04:50:07 STEP 2747 H=2747:00 +CPU= 16.131
  04:50:36 STEP 2748 H=2748:00 +CPU= 28.539
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002748
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002748
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002748
  04:50:52 STEP 2749 H=2749:00 +CPU= 16.137
  04:51:09 STEP 2750 H=2750:00 +CPU= 16.622
  04:51:35 STEP 2751 H=2751:00 +CPU= 25.730
  04:51:51 STEP 2752 H=2752:00 +CPU= 16.623
  04:52:07 STEP 2753 H=2753:00 +CPU= 16.109
  04:52:34 STEP 2754 H=2754:00 +CPU= 26.281
  04:52:50 STEP 2755 H=2755:00 +CPU= 16.130
  04:53:06 STEP 2756 H=2756:00 +CPU= 16.587
  04:53:32 STEP 2757 H=2757:00 +CPU= 25.656
  04:53:49 STEP 2758 H=2758:00 +CPU= 16.592
  04:54:05 STEP 2759 H=2759:00 +CPU= 16.624
  04:54:36 STEP 2760 H=2760:00 +CPU= 30.383
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002760
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002760
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002760
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002736
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002736
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002736
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002748
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002748
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002748
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_112.zip
Uploading the intermediate file: upload_file_112.zip
  04:54:52 STEP 2761 H=2761:00 +CPU= 16.146
Uploading trickle at timestep: 9936000
  04:55:09 STEP 2762 H=2762:00 +CPU= 16.640
  04:55:34 STEP 2763 H=2763:00 +CPU= 25.910
  04:55:51 STEP 2764 H=2764:00 +CPU= 16.633
  04:56:07 STEP 2765 H=2765:00 +CPU= 16.147
  04:56:34 STEP 2766 H=2766:00 +CPU= 26.306
  04:56:50 STEP 2767 H=2767:00 +CPU= 16.126
  04:57:06 STEP 2768 H=2768:00 +CPU= 16.625
  04:57:32 STEP 2769 H=2769:00 +CPU= 25.701
  04:57:49 STEP 2770 H=2770:00 +CPU= 16.640
  04:58:05 STEP 2771 H=2771:00 +CPU= 16.158
  04:58:33 STEP 2772 H=2772:00 +CPU= 28.480
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002772
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002772
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002772
  04:58:50 STEP 2773 H=2773:00 +CPU= 16.120
  04:59:06 STEP 2774 H=2774:00 +CPU= 16.636
  04:59:32 STEP 2775 H=2775:00 +CPU= 25.719
  04:59:49 STEP 2776 H=2776:00 +CPU= 16.597
  05:00:05 STEP 2777 H=2777:00 +CPU= 16.114
  05:00:31 STEP 2778 H=2778:00 +CPU= 26.207
  05:00:47 STEP 2779 H=2779:00 +CPU= 16.109
  05:01:04 STEP 2780 H=2780:00 +CPU= 16.552
  05:01:29 STEP 2781 H=2781:00 +CPU= 25.654
  05:01:46 STEP 2782 H=2782:00 +CPU= 16.527
  05:02:02 STEP 2783 H=2783:00 +CPU= 16.583
  05:02:33 STEP 2784 H=2784:00 +CPU= 30.353
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002784
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002784
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002784
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002760
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002760
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002760
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002772
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002772
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002772
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_113.zip
Uploading the intermediate file: upload_file_113.zip
  05:02:49 STEP 2785 H=2785:00 +CPU= 16.168
  05:03:06 STEP 2786 H=2786:00 +CPU= 16.566
  05:03:31 STEP 2787 H=2787:00 +CPU= 25.880
  05:03:48 STEP 2788 H=2788:00 +CPU= 16.586
  05:04:04 STEP 2789 H=2789:00 +CPU= 16.112
  05:04:30 STEP 2790 H=2790:00 +CPU= 26.276
  05:04:47 STEP 2791 H=2791:00 +CPU= 16.153
  05:05:03 STEP 2792 H=2792:00 +CPU= 16.637
  05:05:29 STEP 2793 H=2793:00 +CPU= 25.719
  05:05:46 STEP 2794 H=2794:00 +CPU= 16.633
  05:06:02 STEP 2795 H=2795:00 +CPU= 16.183
  05:06:30 STEP 2796 H=2796:00 +CPU= 28.485
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002796
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002796
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002796
  05:06:47 STEP 2797 H=2797:00 +CPU= 16.163
  05:07:03 STEP 2798 H=2798:00 +CPU= 16.647
  05:07:29 STEP 2799 H=2799:00 +CPU= 25.693
  05:07:46 STEP 2800 H=2800:00 +CPU= 16.628
  05:08:02 STEP 2801 H=2801:00 +CPU= 16.136
  05:08:28 STEP 2802 H=2802:00 +CPU= 26.254
  05:08:44 STEP 2803 H=2803:00 +CPU= 16.128
  05:09:01 STEP 2804 H=2804:00 +CPU= 16.586
  05:09:26 STEP 2805 H=2805:00 +CPU= 25.661
  05:09:43 STEP 2806 H=2806:00 +CPU= 16.571
  05:10:00 STEP 2807 H=2807:00 +CPU= 16.601
  05:10:30 STEP 2808 H=2808:00 +CPU= 30.385
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002808
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002808
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002808
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002784
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002784
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002784
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002796
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002796
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002796
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_114.zip
Uploading the intermediate file: upload_file_114.zip
  05:10:46 STEP 2809 H=2809:00 +CPU= 16.128
  05:11:03 STEP 2810 H=2810:00 +CPU= 16.615
  05:11:29 STEP 2811 H=2811:00 +CPU= 25.875
  05:11:45 STEP 2812 H=2812:00 +CPU= 16.578
  05:12:01 STEP 2813 H=2813:00 +CPU= 16.123
  05:12:28 STEP 2814 H=2814:00 +CPU= 26.245
  05:12:44 STEP 2815 H=2815:00 +CPU= 16.140
  05:13:01 STEP 2816 H=2816:00 +CPU= 16.640
  05:13:26 STEP 2817 H=2817:00 +CPU= 25.696
  05:13:43 STEP 2818 H=2818:00 +CPU= 16.605
  05:13:59 STEP 2819 H=2819:00 +CPU= 16.155
  05:14:28 STEP 2820 H=2820:00 +CPU= 28.479
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002820
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002820
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002820
  05:14:44 STEP 2821 H=2821:00 +CPU= 16.138
  05:15:00 STEP 2822 H=2822:00 +CPU= 16.617
  05:15:26 STEP 2823 H=2823:00 +CPU= 25.694
  05:15:43 STEP 2824 H=2824:00 +CPU= 16.619
  05:15:59 STEP 2825 H=2825:00 +CPU= 16.129
  05:16:25 STEP 2826 H=2826:00 +CPU= 26.222
  05:16:41 STEP 2827 H=2827:00 +CPU= 16.131
  05:16:58 STEP 2828 H=2828:00 +CPU= 16.611
  05:17:23 STEP 2829 H=2829:00 +CPU= 25.652
  05:17:40 STEP 2830 H=2830:00 +CPU= 16.571
  05:17:57 STEP 2831 H=2831:00 +CPU= 16.606
  05:18:27 STEP 2832 H=2832:00 +CPU= 30.351
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002832
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002832
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002832
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002808
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002808
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002808
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002820
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002820
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002820
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_115.zip
Uploading the intermediate file: upload_file_115.zip
  05:18:43 STEP 2833 H=2833:00 +CPU= 16.176
  05:19:00 STEP 2834 H=2834:00 +CPU= 16.612
  05:19:26 STEP 2835 H=2835:00 +CPU= 25.925
  05:19:42 STEP 2836 H=2836:00 +CPU= 16.606
  05:19:59 STEP 2837 H=2837:00 +CPU= 16.105
  05:20:25 STEP 2838 H=2838:00 +CPU= 26.313
  05:20:41 STEP 2839 H=2839:00 +CPU= 16.168
  05:20:58 STEP 2840 H=2840:00 +CPU= 16.647
  05:21:23 STEP 2841 H=2841:00 +CPU= 25.744
  05:21:40 STEP 2842 H=2842:00 +CPU= 16.630
  05:21:56 STEP 2843 H=2843:00 +CPU= 16.151
  05:22:25 STEP 2844 H=2844:00 +CPU= 28.503
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002844
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002844
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002844
  05:22:41 STEP 2845 H=2845:00 +CPU= 16.172
  05:22:58 STEP 2846 H=2846:00 +CPU= 16.637
  05:23:23 STEP 2847 H=2847:00 +CPU= 25.715
  05:23:40 STEP 2848 H=2848:00 +CPU= 16.609
  05:23:56 STEP 2849 H=2849:00 +CPU= 16.146
  05:24:22 STEP 2850 H=2850:00 +CPU= 26.276
  05:24:39 STEP 2851 H=2851:00 +CPU= 16.129
  05:24:55 STEP 2852 H=2852:00 +CPU= 16.599
  05:25:21 STEP 2853 H=2853:00 +CPU= 25.670
  05:25:37 STEP 2854 H=2854:00 +CPU= 16.577
  05:25:54 STEP 2855 H=2855:00 +CPU= 16.652
  05:26:25 STEP 2856 H=2856:00 +CPU= 30.361
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002856
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002856
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002856
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002832
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002832
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002832
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002844
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002844
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002844
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_116.zip
Uploading the intermediate file: upload_file_116.zip
  05:26:41 STEP 2857 H=2857:00 +CPU= 16.178
  05:26:57 STEP 2858 H=2858:00 +CPU= 16.623
  05:27:23 STEP 2859 H=2859:00 +CPU= 25.926
  05:27:40 STEP 2860 H=2860:00 +CPU= 16.625
  05:27:56 STEP 2861 H=2861:00 +CPU= 16.119
  05:28:22 STEP 2862 H=2862:00 +CPU= 26.318
  05:28:39 STEP 2863 H=2863:00 +CPU= 16.131
  05:28:55 STEP 2864 H=2864:00 +CPU= 16.681
  05:29:21 STEP 2865 H=2865:00 +CPU= 25.724
  05:29:38 STEP 2866 H=2866:00 +CPU= 16.604
  05:29:54 STEP 2867 H=2867:00 +CPU= 16.125
  05:30:22 STEP 2868 H=2868:00 +CPU= 28.459
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002868
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002868
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002868
  05:30:38 STEP 2869 H=2869:00 +CPU= 16.150
  05:30:55 STEP 2870 H=2870:00 +CPU= 16.614
  05:31:21 STEP 2871 H=2871:00 +CPU= 25.761
  05:31:37 STEP 2872 H=2872:00 +CPU= 16.617
  05:31:54 STEP 2873 H=2873:00 +CPU= 16.135
  05:32:20 STEP 2874 H=2874:00 +CPU= 26.251
  05:32:36 STEP 2875 H=2875:00 +CPU= 16.115
  05:32:53 STEP 2876 H=2876:00 +CPU= 16.584
  05:33:18 STEP 2877 H=2877:00 +CPU= 25.636
  05:33:35 STEP 2878 H=2878:00 +CPU= 16.572
  05:33:51 STEP 2879 H=2879:00 +CPU= 16.611
  05:34:22 STEP 2880 H=2880:00 +CPU= 30.326
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002880
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002880
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002880
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002856
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002856
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002856
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002868
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002868
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002868
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_117.zip
Uploading the intermediate file: upload_file_117.zip
  05:34:38 STEP 2881 H=2881:00 +CPU= 16.117
  05:34:55 STEP 2882 H=2882:00 +CPU= 16.661
  05:35:21 STEP 2883 H=2883:00 +CPU= 25.889
  05:35:37 STEP 2884 H=2884:00 +CPU= 16.601
  05:35:53 STEP 2885 H=2885:00 +CPU= 16.193
  05:36:20 STEP 2886 H=2886:00 +CPU= 26.240
  05:36:36 STEP 2887 H=2887:00 +CPU= 16.146
  05:36:52 STEP 2888 H=2888:00 +CPU= 16.588
  05:37:18 STEP 2889 H=2889:00 +CPU= 25.718
  05:37:35 STEP 2890 H=2890:00 +CPU= 16.580
  05:37:51 STEP 2891 H=2891:00 +CPU= 16.157
  05:38:19 STEP 2892 H=2892:00 +CPU= 28.483
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002892
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002892
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002892
  05:38:36 STEP 2893 H=2893:00 +CPU= 16.122
  05:38:52 STEP 2894 H=2894:00 +CPU= 16.610
  05:39:18 STEP 2895 H=2895:00 +CPU= 25.723
  05:39:35 STEP 2896 H=2896:00 +CPU= 16.594
  05:39:51 STEP 2897 H=2897:00 +CPU= 16.108
  05:40:17 STEP 2898 H=2898:00 +CPU= 26.202
  05:40:33 STEP 2899 H=2899:00 +CPU= 16.126
  05:40:50 STEP 2900 H=2900:00 +CPU= 16.630
  05:41:15 STEP 2901 H=2901:00 +CPU= 25.719
  05:41:32 STEP 2902 H=2902:00 +CPU= 16.567
  05:41:49 STEP 2903 H=2903:00 +CPU= 16.596
  05:42:19 STEP 2904 H=2904:00 +CPU= 30.335
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002904
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002904
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002904
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002880
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002880
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002880
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002892
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002892
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002892
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_118.zip
Uploading the intermediate file: upload_file_118.zip
  05:42:35 STEP 2905 H=2905:00 +CPU= 16.151
  05:42:52 STEP 2906 H=2906:00 +CPU= 16.571
  05:43:18 STEP 2907 H=2907:00 +CPU= 25.934
  05:43:34 STEP 2908 H=2908:00 +CPU= 16.592
  05:43:50 STEP 2909 H=2909:00 +CPU= 16.104
  05:44:17 STEP 2910 H=2910:00 +CPU= 26.325
  05:44:33 STEP 2911 H=2911:00 +CPU= 16.157
  05:44:50 STEP 2912 H=2912:00 +CPU= 16.619
  05:45:15 STEP 2913 H=2913:00 +CPU= 25.728
  05:45:32 STEP 2914 H=2914:00 +CPU= 16.598
  05:45:48 STEP 2915 H=2915:00 +CPU= 16.171
  05:46:17 STEP 2916 H=2916:00 +CPU= 28.557
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002916
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002916
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002916
  05:46:33 STEP 2917 H=2917:00 +CPU= 16.168
  05:46:50 STEP 2918 H=2918:00 +CPU= 16.645
  05:47:15 STEP 2919 H=2919:00 +CPU= 25.753
  05:47:32 STEP 2920 H=2920:00 +CPU= 16.599
  05:47:48 STEP 2921 H=2921:00 +CPU= 16.105
  05:48:14 STEP 2922 H=2922:00 +CPU= 26.250
  05:48:30 STEP 2923 H=2923:00 +CPU= 16.104
  05:48:47 STEP 2924 H=2924:00 +CPU= 16.591
  05:49:13 STEP 2925 H=2925:00 +CPU= 25.670
  05:49:29 STEP 2926 H=2926:00 +CPU= 16.560
  05:49:46 STEP 2927 H=2927:00 +CPU= 16.582
  05:50:16 STEP 2928 H=2928:00 +CPU= 30.312
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002928
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002928
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002928
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002904
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002904
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002904
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002916
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002916
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002916
Zipping up the intermediate file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_119.zip
Uploading the intermediate file: upload_file_119.zip
  05:50:32 STEP 2929 H=2929:00 +CPU= 16.150
  05:50:49 STEP 2930 H=2930:00 +CPU= 16.661
  05:51:15 STEP 2931 H=2931:00 +CPU= 25.857
  05:51:32 STEP 2932 H=2932:00 +CPU= 16.580
  05:51:48 STEP 2933 H=2933:00 +CPU= 16.131
  05:52:14 STEP 2934 H=2934:00 +CPU= 26.291
  05:52:30 STEP 2935 H=2935:00 +CPU= 16.143
  05:52:47 STEP 2936 H=2936:00 +CPU= 16.629
  05:53:13 STEP 2937 H=2937:00 +CPU= 25.727
  05:53:29 STEP 2938 H=2938:00 +CPU= 16.585
  05:53:45 STEP 2939 H=2939:00 +CPU= 16.116
  05:54:14 STEP 2940 H=2940:00 +CPU= 28.464
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002940
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002940
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002940
  05:54:30 STEP 2941 H=2941:00 +CPU= 16.128
  05:54:47 STEP 2942 H=2942:00 +CPU= 16.617
  05:55:12 STEP 2943 H=2943:00 +CPU= 25.709
  05:55:29 STEP 2944 H=2944:00 +CPU= 16.596
  05:55:45 STEP 2945 H=2945:00 +CPU= 16.100
  05:56:11 STEP 2946 H=2946:00 +CPU= 26.274
  05:56:27 STEP 2947 H=2947:00 +CPU= 16.104
  05:56:44 STEP 2948 H=2948:00 +CPU= 16.544
  05:57:10 STEP 2949 H=2949:00 +CPU= 25.694
  05:57:26 STEP 2950 H=2950:00 +CPU= 16.550
  05:57:43 STEP 2951 H=2951:00 +CPU= 16.579
  05:58:12 STEP 2952 H=2952:00 +CPU= 29.500
The child process terminated with status: 0
Moving to projects directory: /home/mdmann/slots/0/ICMGGhq0f+002952
Moving to projects directory: /home/mdmann/slots/0/ICMSHhq0f+002952
Moving to projects directory: /home/mdmann/slots/0/ICMUAhq0f+002952
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002928
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002928
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002940
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002928
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002940
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMSHhq0f+002952
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002940
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMGGhq0f+002952
Adding to the zip: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_12172565/ICMUAhq0f+002952
Zipping up the final file: /home/mdmann/projects/climateprediction.net/oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_120.zip
Uploading the final file: upload_file_120.zip
Uploading trickle at timestep: 10623600
06:01:36 (5199): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_121.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_ps_0921_1986050100_123_955_12172565_0_r1200163672_122.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Mar 2024 16:30:39 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 10,623,600 59,626 0.0056
01 Mar 2024 15:29:47 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 9,936,000 55,814 0.0056
01 Mar 2024 14:29:31 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 9,072,000 51,055 0.0056
01 Mar 2024 12:27:29 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 8,208,000 46,287 0.0056
01 Mar 2024 11:26:14 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 7,344,000 41,519 0.0057
01 Mar 2024 10:25:21 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 6,480,000 36,740 0.0057
01 Mar 2024 08:24:39 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 5,616,000 31,972 0.0057
01 Mar 2024 07:23:43 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 4,752,000 27,204 0.0057
01 Mar 2024 06:22:46 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 3,888,000 22,427 0.0058
01 Mar 2024 04:21:58 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 3,024,000 17,649 0.0058
01 Mar 2024 03:20:59 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 2,160,000 12,882 0.0060
01 Mar 2024 02:20:48 1514362 22257056 oifs_43r3_ps_0921_1986050100_123_955_12172565_0 1,296,000 8,115 0.0063


©2024 climateprediction.net