climateprediction.net home page
Task 22283359

Task 22283359

Name oifs_43r3_ps_0683_2011050100_123_980_12197327_0
Workunit 12197327
Created 24 Dec 2022, 2:59:51 UTC
Sent 18 Jan 2023, 21:01:06 UTC
Report deadline 19 Mar 2023, 21:01:06 UTC
Received 19 Jan 2023, 8:08:30 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1535413
Run time 10 hours 59 min 57 sec
CPU time 10 hours 49 min 55 sec
Validate state Invalid
Credit 13,910.40
Device peak FLOPS 7.91 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,405.31 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,038.40 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Process still present 5 min after writing finish file; aborting</message>
<stderr_txt>
 /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002256
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002256
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002268
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002268
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002268
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_94.zip
Uploading the intermediate file: upload_file_94.zip
  00:25:34 STEP 2281 H=2281:00 +CPU= 10.663
  00:25:45 STEP 2282 H=2282:00 +CPU= 10.946
Uploading trickle at timestep: 8208000
  00:26:02 STEP 2283 H=2283:00 +CPU= 17.232
  00:26:13 STEP 2284 H=2284:00 +CPU= 10.804
  00:26:23 STEP 2285 H=2285:00 +CPU= 10.731
  00:26:41 STEP 2286 H=2286:00 +CPU= 17.438
  00:26:51 STEP 2287 H=2287:00 +CPU= 10.676
  00:27:03 STEP 2288 H=2288:00 +CPU= 11.096
  00:27:20 STEP 2289 H=2289:00 +CPU= 17.139
  00:27:31 STEP 2290 H=2290:00 +CPU= 10.914
  00:27:42 STEP 2291 H=2291:00 +CPU= 10.874
  00:28:01 STEP 2292 H=2292:00 +CPU= 19.121
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002292
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002292
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002292
  00:28:11 STEP 2293 H=2293:00 +CPU= 10.541
  00:28:22 STEP 2294 H=2294:00 +CPU= 10.685
  00:28:39 STEP 2295 H=2295:00 +CPU= 17.259
  00:28:50 STEP 2296 H=2296:00 +CPU= 11.153
  00:29:01 STEP 2297 H=2297:00 +CPU= 10.638
  00:29:18 STEP 2298 H=2298:00 +CPU= 17.460
  00:29:29 STEP 2299 H=2299:00 +CPU= 10.687
  00:29:41 STEP 2300 H=2300:00 +CPU= 11.300
  00:29:58 STEP 2301 H=2301:00 +CPU= 17.073
  00:30:09 STEP 2302 H=2302:00 +CPU= 10.930
  00:30:19 STEP 2303 H=2303:00 +CPU= 10.964
  00:30:40 STEP 2304 H=2304:00 +CPU= 20.589
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002304
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002304
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002304
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002280
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002280
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002280
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002292
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002292
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002292
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_95.zip
Uploading the intermediate file: upload_file_95.zip
  00:30:51 STEP 2305 H=2305:00 +CPU= 10.409
  00:31:01 STEP 2306 H=2306:00 +CPU= 10.815
  00:31:18 STEP 2307 H=2307:00 +CPU= 17.085
  00:31:29 STEP 2308 H=2308:00 +CPU= 10.891
  00:31:40 STEP 2309 H=2309:00 +CPU= 10.712
  00:31:58 STEP 2310 H=2310:00 +CPU= 17.428
  00:32:08 STEP 2311 H=2311:00 +CPU= 10.672
  00:32:20 STEP 2312 H=2312:00 +CPU= 11.266
  00:32:37 STEP 2313 H=2313:00 +CPU= 17.318
  00:32:48 STEP 2314 H=2314:00 +CPU= 10.799
  00:32:58 STEP 2315 H=2315:00 +CPU= 10.605
  00:33:17 STEP 2316 H=2316:00 +CPU= 19.034
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002316
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002316
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002316
  00:33:28 STEP 2317 H=2317:00 +CPU= 10.772
  00:33:39 STEP 2318 H=2318:00 +CPU= 10.811
  00:33:56 STEP 2319 H=2319:00 +CPU= 16.818
  00:34:07 STEP 2320 H=2320:00 +CPU= 11.037
  00:34:18 STEP 2321 H=2321:00 +CPU= 10.803
  00:34:35 STEP 2322 H=2322:00 +CPU= 17.620
  00:34:46 STEP 2323 H=2323:00 +CPU= 10.646
  00:34:57 STEP 2324 H=2324:00 +CPU= 11.101
  00:35:14 STEP 2325 H=2325:00 +CPU= 17.238
  00:35:25 STEP 2326 H=2326:00 +CPU= 11.098
  00:35:36 STEP 2327 H=2327:00 +CPU= 11.000
  00:35:57 STEP 2328 H=2328:00 +CPU= 20.131
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002328
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002328
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002328
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002304
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002304
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002304
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002316
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002316
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002316
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_96.zip
Uploading the intermediate file: upload_file_96.zip
  00:36:07 STEP 2329 H=2329:00 +CPU= 10.691
  00:36:18 STEP 2330 H=2330:00 +CPU= 11.019
  00:36:35 STEP 2331 H=2331:00 +CPU= 17.059
  00:36:46 STEP 2332 H=2332:00 +CPU= 11.005
  00:36:57 STEP 2333 H=2333:00 +CPU= 10.989
  00:37:15 STEP 2334 H=2334:00 +CPU= 17.544
  00:37:26 STEP 2335 H=2335:00 +CPU= 10.752
  00:37:37 STEP 2336 H=2336:00 +CPU= 11.117
  00:37:54 STEP 2337 H=2337:00 +CPU= 17.550
  00:38:05 STEP 2338 H=2338:00 +CPU= 10.931
  00:38:16 STEP 2339 H=2339:00 +CPU= 10.839
  00:38:35 STEP 2340 H=2340:00 +CPU= 19.013
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002340
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002340
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002340
  00:38:46 STEP 2341 H=2341:00 +CPU= 10.581
  00:38:57 STEP 2342 H=2342:00 +CPU= 10.879
  00:39:14 STEP 2343 H=2343:00 +CPU= 17.247
  00:39:25 STEP 2344 H=2344:00 +CPU= 10.929
  00:39:36 STEP 2345 H=2345:00 +CPU= 10.752
  00:39:53 STEP 2346 H=2346:00 +CPU= 17.535
  00:40:04 STEP 2347 H=2347:00 +CPU= 10.694
  00:40:15 STEP 2348 H=2348:00 +CPU= 11.037
  00:40:32 STEP 2349 H=2349:00 +CPU= 16.917
  00:40:43 STEP 2350 H=2350:00 +CPU= 10.844
  00:40:54 STEP 2351 H=2351:00 +CPU= 10.999
  00:41:14 STEP 2352 H=2352:00 +CPU= 20.770
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002352
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002352
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002352
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002328
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002328
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002328
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002340
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002340
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002340
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_97.zip
Uploading the intermediate file: upload_file_97.zip
  00:41:25 STEP 2353 H=2353:00 +CPU= 10.683
  00:41:36 STEP 2354 H=2354:00 +CPU= 11.106
  00:41:53 STEP 2355 H=2355:00 +CPU= 17.010
  00:42:04 STEP 2356 H=2356:00 +CPU= 11.134
  00:42:15 STEP 2357 H=2357:00 +CPU= 10.872
  00:42:33 STEP 2358 H=2358:00 +CPU= 17.426
  00:42:43 STEP 2359 H=2359:00 +CPU= 10.599
  00:42:54 STEP 2360 H=2360:00 +CPU= 11.082
  00:43:12 STEP 2361 H=2361:00 +CPU= 17.374
  00:43:23 STEP 2362 H=2362:00 +CPU= 11.063
  00:43:34 STEP 2363 H=2363:00 +CPU= 10.593
  00:43:53 STEP 2364 H=2364:00 +CPU= 19.080
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002364
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002364
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002364
  00:44:03 STEP 2365 H=2365:00 +CPU= 10.824
  00:44:14 STEP 2366 H=2366:00 +CPU= 10.952
  00:44:32 STEP 2367 H=2367:00 +CPU= 17.161
  00:44:43 STEP 2368 H=2368:00 +CPU= 11.119
  00:44:53 STEP 2369 H=2369:00 +CPU= 10.576
  00:45:11 STEP 2370 H=2370:00 +CPU= 17.437
  00:45:22 STEP 2371 H=2371:00 +CPU= 10.988
  00:45:33 STEP 2372 H=2372:00 +CPU= 11.002
  00:45:50 STEP 2373 H=2373:00 +CPU= 17.172
  00:46:01 STEP 2374 H=2374:00 +CPU= 11.078
  00:46:12 STEP 2375 H=2375:00 +CPU= 11.355
  00:46:33 STEP 2376 H=2376:00 +CPU= 20.571
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002376
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002376
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002376
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002352
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002352
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002352
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002364
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002364
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002364
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_98.zip
Uploading the intermediate file: upload_file_98.zip
  00:46:44 STEP 2377 H=2377:00 +CPU= 10.664
  00:46:55 STEP 2378 H=2378:00 +CPU= 10.986
  00:47:12 STEP 2379 H=2379:00 +CPU= 17.101
  00:47:23 STEP 2380 H=2380:00 +CPU= 10.895
  00:47:33 STEP 2381 H=2381:00 +CPU= 10.761
  00:47:51 STEP 2382 H=2382:00 +CPU= 17.661
  00:48:02 STEP 2383 H=2383:00 +CPU= 10.941
  00:48:13 STEP 2384 H=2384:00 +CPU= 11.124
  00:48:30 STEP 2385 H=2385:00 +CPU= 17.040
  00:48:41 STEP 2386 H=2386:00 +CPU= 11.027
  00:48:52 STEP 2387 H=2387:00 +CPU= 10.925
  00:49:12 STEP 2388 H=2388:00 +CPU= 19.310
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002388
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002388
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002388
  00:49:22 STEP 2389 H=2389:00 +CPU= 10.606
  00:49:33 STEP 2390 H=2390:00 +CPU= 11.195
  00:49:50 STEP 2391 H=2391:00 +CPU= 16.883
  00:50:01 STEP 2392 H=2392:00 +CPU= 10.883
  00:50:12 STEP 2393 H=2393:00 +CPU= 10.799
  00:50:29 STEP 2394 H=2394:00 +CPU= 17.466
  00:50:40 STEP 2395 H=2395:00 +CPU= 10.695
  00:50:51 STEP 2396 H=2396:00 +CPU= 11.070
  00:51:08 STEP 2397 H=2397:00 +CPU= 17.166
  00:51:19 STEP 2398 H=2398:00 +CPU= 10.893
  00:51:31 STEP 2399 H=2399:00 +CPU= 11.385
  00:51:51 STEP 2400 H=2400:00 +CPU= 20.367
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002400
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002400
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002400
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002376
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002376
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002376
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002388
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002388
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002388
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_99.zip
Uploading the intermediate file: upload_file_99.zip
  00:52:02 STEP 2401 H=2401:00 +CPU= 10.713
  00:52:13 STEP 2402 H=2402:00 +CPU= 11.089
  00:52:30 STEP 2403 H=2403:00 +CPU= 17.182
  00:52:41 STEP 2404 H=2404:00 +CPU= 11.093
  00:52:52 STEP 2405 H=2405:00 +CPU= 10.512
  00:53:09 STEP 2406 H=2406:00 +CPU= 17.683
  00:53:20 STEP 2407 H=2407:00 +CPU= 10.796
  00:53:31 STEP 2408 H=2408:00 +CPU= 11.029
  00:53:49 STEP 2409 H=2409:00 +CPU= 17.180
  00:54:00 STEP 2410 H=2410:00 +CPU= 11.120
  00:54:11 STEP 2411 H=2411:00 +CPU= 10.903
  00:54:30 STEP 2412 H=2412:00 +CPU= 18.900
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002412
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002412
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002412
  00:54:40 STEP 2413 H=2413:00 +CPU= 10.864
  00:54:52 STEP 2414 H=2414:00 +CPU= 11.268
  00:55:09 STEP 2415 H=2415:00 +CPU= 16.885
  00:55:20 STEP 2416 H=2416:00 +CPU= 11.271
  00:55:31 STEP 2417 H=2417:00 +CPU= 10.793
  00:55:48 STEP 2418 H=2418:00 +CPU= 17.830
  00:55:59 STEP 2419 H=2419:00 +CPU= 10.781
  00:56:10 STEP 2420 H=2420:00 +CPU= 11.093
  00:56:28 STEP 2421 H=2421:00 +CPU= 17.187
  00:56:39 STEP 2422 H=2422:00 +CPU= 11.107
  00:56:50 STEP 2423 H=2423:00 +CPU= 11.319
  00:57:11 STEP 2424 H=2424:00 +CPU= 20.562
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002424
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002424
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002424
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002400
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002400
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002400
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002412
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002412
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002412
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_100.zip
Uploading the intermediate file: upload_file_100.zip
  00:57:21 STEP 2425 H=2425:00 +CPU= 10.637
  00:57:32 STEP 2426 H=2426:00 +CPU= 10.687
  00:57:49 STEP 2427 H=2427:00 +CPU= 17.112
  00:58:00 STEP 2428 H=2428:00 +CPU= 10.945
  00:58:11 STEP 2429 H=2429:00 +CPU= 10.964
  00:58:29 STEP 2430 H=2430:00 +CPU= 17.526
  00:58:39 STEP 2431 H=2431:00 +CPU= 10.650
  00:58:50 STEP 2432 H=2432:00 +CPU= 11.179
  00:59:07 STEP 2433 H=2433:00 +CPU= 16.900
  00:59:18 STEP 2434 H=2434:00 +CPU= 11.142
  00:59:29 STEP 2435 H=2435:00 +CPU= 10.970
  00:59:49 STEP 2436 H=2436:00 +CPU= 19.323
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002436
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002436
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002436
  00:59:59 STEP 2437 H=2437:00 +CPU= 10.612
  01:00:10 STEP 2438 H=2438:00 +CPU= 10.850
  01:00:27 STEP 2439 H=2439:00 +CPU= 16.949
  01:00:38 STEP 2440 H=2440:00 +CPU= 10.915
  01:00:49 STEP 2441 H=2441:00 +CPU= 10.766
  01:01:06 STEP 2442 H=2442:00 +CPU= 17.538
  01:01:17 STEP 2443 H=2443:00 +CPU= 10.685
  01:01:28 STEP 2444 H=2444:00 +CPU= 11.151
  01:01:46 STEP 2445 H=2445:00 +CPU= 17.235
  01:01:56 STEP 2446 H=2446:00 +CPU= 10.868
  01:02:08 STEP 2447 H=2447:00 +CPU= 11.374
  01:02:28 STEP 2448 H=2448:00 +CPU= 20.549
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002448
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002448
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002448
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002424
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002424
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002424
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002436
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002436
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002436
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_101.zip
Uploading the intermediate file: upload_file_101.zip
  01:02:39 STEP 2449 H=2449:00 +CPU= 10.675
  01:02:50 STEP 2450 H=2450:00 +CPU= 10.797
  01:03:07 STEP 2451 H=2451:00 +CPU= 17.192
  01:03:18 STEP 2452 H=2452:00 +CPU= 11.008
  01:03:29 STEP 2453 H=2453:00 +CPU= 10.922
  01:03:47 STEP 2454 H=2454:00 +CPU= 17.515
  01:03:58 STEP 2455 H=2455:00 +CPU= 10.900
  01:04:09 STEP 2456 H=2456:00 +CPU= 11.145
  01:04:26 STEP 2457 H=2457:00 +CPU= 17.200
  01:04:37 STEP 2458 H=2458:00 +CPU= 11.015
  01:04:48 STEP 2459 H=2459:00 +CPU= 10.809
  01:05:07 STEP 2460 H=2460:00 +CPU= 19.092
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002460
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002460
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002460
  01:05:17 STEP 2461 H=2461:00 +CPU= 10.660
  01:05:29 STEP 2462 H=2462:00 +CPU= 11.054
  01:05:45 STEP 2463 H=2463:00 +CPU= 16.807
  01:05:56 STEP 2464 H=2464:00 +CPU= 10.932
  01:06:07 STEP 2465 H=2465:00 +CPU= 10.967
  01:06:25 STEP 2466 H=2466:00 +CPU= 17.361
  01:06:35 STEP 2467 H=2467:00 +CPU= 10.698
  01:06:46 STEP 2468 H=2468:00 +CPU= 10.975
  01:07:03 STEP 2469 H=2469:00 +CPU= 17.026
  01:07:14 STEP 2470 H=2470:00 +CPU= 10.804
  01:07:25 STEP 2471 H=2471:00 +CPU= 11.038
  01:07:46 STEP 2472 H=2472:00 +CPU= 20.456
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002472
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002472
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002472
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002448
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002448
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002448
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002460
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002460
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002460
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_102.zip
Uploading the intermediate file: upload_file_102.zip
  01:07:56 STEP 2473 H=2473:00 +CPU= 10.673
  01:08:08 STEP 2474 H=2474:00 +CPU= 11.203
  01:08:25 STEP 2475 H=2475:00 +CPU= 17.048
  01:08:36 STEP 2476 H=2476:00 +CPU= 11.161
  01:08:47 STEP 2477 H=2477:00 +CPU= 10.954
  01:09:05 STEP 2478 H=2478:00 +CPU= 17.659
  01:09:15 STEP 2479 H=2479:00 +CPU= 10.718
  01:09:26 STEP 2480 H=2480:00 +CPU= 10.945
  01:09:43 STEP 2481 H=2481:00 +CPU= 17.211
  01:09:55 STEP 2482 H=2482:00 +CPU= 11.247
  01:10:05 STEP 2483 H=2483:00 +CPU= 10.692
  01:10:24 STEP 2484 H=2484:00 +CPU= 18.966
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002484
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002484
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002484
  01:10:35 STEP 2485 H=2485:00 +CPU= 10.693
  01:10:46 STEP 2486 H=2486:00 +CPU= 11.110
  01:11:03 STEP 2487 H=2487:00 +CPU= 16.984
  01:11:14 STEP 2488 H=2488:00 +CPU= 10.865
  01:11:25 STEP 2489 H=2489:00 +CPU= 10.818
  01:11:42 STEP 2490 H=2490:00 +CPU= 17.343
  01:11:53 STEP 2491 H=2491:00 +CPU= 10.579
  01:12:04 STEP 2492 H=2492:00 +CPU= 10.864
  01:12:21 STEP 2493 H=2493:00 +CPU= 17.196
  01:12:32 STEP 2494 H=2494:00 +CPU= 10.935
  01:12:43 STEP 2495 H=2495:00 +CPU= 10.787
  01:13:03 STEP 2496 H=2496:00 +CPU= 20.501
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002496
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002496
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002496
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002472
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002472
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002472
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002484
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002484
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002484
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_103.zip
Uploading the intermediate file: upload_file_103.zip
  01:13:14 STEP 2497 H=2497:00 +CPU= 10.910
  01:13:25 STEP 2498 H=2498:00 +CPU= 10.853
  01:13:42 STEP 2499 H=2499:00 +CPU= 17.139
  01:13:53 STEP 2500 H=2500:00 +CPU= 11.025
  01:14:04 STEP 2501 H=2501:00 +CPU= 10.863
  01:14:22 STEP 2502 H=2502:00 +CPU= 17.605
  01:14:32 STEP 2503 H=2503:00 +CPU= 10.551
  01:14:43 STEP 2504 H=2504:00 +CPU= 11.173
  01:15:01 STEP 2505 H=2505:00 +CPU= 17.175
  01:15:12 STEP 2506 H=2506:00 +CPU= 10.999
  01:15:22 STEP 2507 H=2507:00 +CPU= 10.720
  01:15:42 STEP 2508 H=2508:00 +CPU= 19.188
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002508
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002508
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002508
  01:15:52 STEP 2509 H=2509:00 +CPU= 10.786
  01:16:03 STEP 2510 H=2510:00 +CPU= 11.059
  01:16:21 STEP 2511 H=2511:00 +CPU= 17.161
  01:16:32 STEP 2512 H=2512:00 +CPU= 11.014
  01:16:42 STEP 2513 H=2513:00 +CPU= 10.779
  01:17:00 STEP 2514 H=2514:00 +CPU= 17.278
  01:17:11 STEP 2515 H=2515:00 +CPU= 10.817
  01:17:22 STEP 2516 H=2516:00 +CPU= 11.122
  01:17:39 STEP 2517 H=2517:00 +CPU= 17.070
  01:17:50 STEP 2518 H=2518:00 +CPU= 11.020
  01:18:01 STEP 2519 H=2519:00 +CPU= 11.045
  01:18:21 STEP 2520 H=2520:00 +CPU= 20.428
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002520
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002520
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002520
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002496
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002496
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002496
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002508
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002508
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002508
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_104.zip
Uploading the intermediate file: upload_file_104.zip
  01:18:32 STEP 2521 H=2521:00 +CPU= 10.764
  01:18:43 STEP 2522 H=2522:00 +CPU= 11.311
Uploading trickle at timestep: 9072000
  01:19:01 STEP 2523 H=2523:00 +CPU= 17.250
  01:19:12 STEP 2524 H=2524:00 +CPU= 10.953
  01:19:23 STEP 2525 H=2525:00 +CPU= 10.855
  01:19:40 STEP 2526 H=2526:00 +CPU= 17.390
  01:19:51 STEP 2527 H=2527:00 +CPU= 10.686
  01:20:02 STEP 2528 H=2528:00 +CPU= 11.048
  01:20:19 STEP 2529 H=2529:00 +CPU= 17.167
  01:20:30 STEP 2530 H=2530:00 +CPU= 10.958
  01:20:41 STEP 2531 H=2531:00 +CPU= 10.721
  01:21:00 STEP 2532 H=2532:00 +CPU= 19.057
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002532
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002532
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002532
  01:21:10 STEP 2533 H=2533:00 +CPU= 10.740
  01:21:22 STEP 2534 H=2534:00 +CPU= 11.195
  01:21:39 STEP 2535 H=2535:00 +CPU= 17.347
  01:21:50 STEP 2536 H=2536:00 +CPU= 11.015
  01:22:01 STEP 2537 H=2537:00 +CPU= 10.781
  01:22:18 STEP 2538 H=2538:00 +CPU= 17.593
  01:22:29 STEP 2539 H=2539:00 +CPU= 10.614
  01:22:40 STEP 2540 H=2540:00 +CPU= 11.051
  01:22:57 STEP 2541 H=2541:00 +CPU= 17.059
  01:23:08 STEP 2542 H=2542:00 +CPU= 11.103
  01:23:19 STEP 2543 H=2543:00 +CPU= 11.047
  01:23:40 STEP 2544 H=2544:00 +CPU= 20.717
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002544
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002544
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002544
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002520
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002520
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002520
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002532
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002532
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002532
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_105.zip
Uploading the intermediate file: upload_file_105.zip
  01:23:51 STEP 2545 H=2545:00 +CPU= 10.679
  01:24:01 STEP 2546 H=2546:00 +CPU= 10.699
  01:24:19 STEP 2547 H=2547:00 +CPU= 17.521
  01:24:30 STEP 2548 H=2548:00 +CPU= 10.914
  01:24:41 STEP 2549 H=2549:00 +CPU= 10.676
  01:24:58 STEP 2550 H=2550:00 +CPU= 17.738
  01:25:09 STEP 2551 H=2551:00 +CPU= 10.623
  01:25:20 STEP 2552 H=2552:00 +CPU= 11.033
  01:25:37 STEP 2553 H=2553:00 +CPU= 17.101
  01:25:48 STEP 2554 H=2554:00 +CPU= 10.777
  01:25:59 STEP 2555 H=2555:00 +CPU= 10.588
  01:26:17 STEP 2556 H=2556:00 +CPU= 18.910
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002556
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002556
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002556
  01:26:28 STEP 2557 H=2557:00 +CPU= 10.511
  01:26:39 STEP 2558 H=2558:00 +CPU= 11.109
  01:26:56 STEP 2559 H=2559:00 +CPU= 17.170
  01:27:07 STEP 2560 H=2560:00 +CPU= 11.082
  01:27:18 STEP 2561 H=2561:00 +CPU= 10.633
  01:27:35 STEP 2562 H=2562:00 +CPU= 17.259
  01:27:46 STEP 2563 H=2563:00 +CPU= 10.778
  01:27:57 STEP 2564 H=2564:00 +CPU= 10.790
  01:28:14 STEP 2565 H=2565:00 +CPU= 17.181
  01:28:25 STEP 2566 H=2566:00 +CPU= 10.992
  01:28:36 STEP 2567 H=2567:00 +CPU= 11.000
  01:28:57 STEP 2568 H=2568:00 +CPU= 20.471
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002568
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002568
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002568
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002544
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002544
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002544
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002556
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002556
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002556
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_106.zip
Uploading the intermediate file: upload_file_106.zip
  01:29:07 STEP 2569 H=2569:00 +CPU= 10.841
  01:29:18 STEP 2570 H=2570:00 +CPU= 10.973
  01:29:35 STEP 2571 H=2571:00 +CPU= 16.986
  01:29:47 STEP 2572 H=2572:00 +CPU= 11.244
  01:29:57 STEP 2573 H=2573:00 +CPU= 10.667
  01:30:15 STEP 2574 H=2574:00 +CPU= 17.360
  01:30:26 STEP 2575 H=2575:00 +CPU= 10.837
  01:30:37 STEP 2576 H=2576:00 +CPU= 11.027
  01:30:54 STEP 2577 H=2577:00 +CPU= 17.106
  01:31:05 STEP 2578 H=2578:00 +CPU= 11.090
  01:31:16 STEP 2579 H=2579:00 +CPU= 10.969
  01:31:35 STEP 2580 H=2580:00 +CPU= 19.104
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002580
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002580
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002580
  01:31:46 STEP 2581 H=2581:00 +CPU= 10.710
  01:31:57 STEP 2582 H=2582:00 +CPU= 11.312
  01:32:14 STEP 2583 H=2583:00 +CPU= 17.243
  01:32:25 STEP 2584 H=2584:00 +CPU= 10.834
  01:32:36 STEP 2585 H=2585:00 +CPU= 10.790
  01:32:53 STEP 2586 H=2586:00 +CPU= 17.551
  01:33:04 STEP 2587 H=2587:00 +CPU= 10.672
  01:33:15 STEP 2588 H=2588:00 +CPU= 10.728
  01:33:32 STEP 2589 H=2589:00 +CPU= 17.248
  01:33:43 STEP 2590 H=2590:00 +CPU= 10.900
  01:33:54 STEP 2591 H=2591:00 +CPU= 11.363
  01:34:15 STEP 2592 H=2592:00 +CPU= 20.683
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002592
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002592
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002592
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002568
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002568
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002568
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002580
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002580
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002580
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_107.zip
Uploading the intermediate file: upload_file_107.zip
  01:34:26 STEP 2593 H=2593:00 +CPU= 10.758
  01:34:37 STEP 2594 H=2594:00 +CPU= 11.156
  01:34:54 STEP 2595 H=2595:00 +CPU= 17.223
  01:35:05 STEP 2596 H=2596:00 +CPU= 11.054
  01:35:16 STEP 2597 H=2597:00 +CPU= 10.783
  01:35:34 STEP 2598 H=2598:00 +CPU= 17.422
  01:35:44 STEP 2599 H=2599:00 +CPU= 10.434
  01:35:56 STEP 2600 H=2600:00 +CPU= 11.392
  01:36:13 STEP 2601 H=2601:00 +CPU= 17.265
  01:36:24 STEP 2602 H=2602:00 +CPU= 10.931
  01:36:35 STEP 2603 H=2603:00 +CPU= 10.885
  01:36:54 STEP 2604 H=2604:00 +CPU= 18.841
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002604
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002604
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002604
  01:37:04 STEP 2605 H=2605:00 +CPU= 10.638
  01:37:15 STEP 2606 H=2606:00 +CPU= 11.232
  01:37:32 STEP 2607 H=2607:00 +CPU= 17.062
  01:37:43 STEP 2608 H=2608:00 +CPU= 10.931
  01:37:54 STEP 2609 H=2609:00 +CPU= 10.790
  01:38:12 STEP 2610 H=2610:00 +CPU= 17.463
  01:38:22 STEP 2611 H=2611:00 +CPU= 10.605
  01:38:33 STEP 2612 H=2612:00 +CPU= 10.957
  01:38:50 STEP 2613 H=2613:00 +CPU= 17.021
  01:39:01 STEP 2614 H=2614:00 +CPU= 10.917
  01:39:12 STEP 2615 H=2615:00 +CPU= 11.156
  01:39:33 STEP 2616 H=2616:00 +CPU= 20.485
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002616
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002616
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002616
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002592
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002592
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002592
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002604
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002604
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002604
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_108.zip
Uploading the intermediate file: upload_file_108.zip
  01:39:44 STEP 2617 H=2617:00 +CPU= 10.724
  01:39:55 STEP 2618 H=2618:00 +CPU= 11.178
  01:40:12 STEP 2619 H=2619:00 +CPU= 17.167
  01:40:23 STEP 2620 H=2620:00 +CPU= 10.990
  01:40:34 STEP 2621 H=2621:00 +CPU= 10.810
  01:40:52 STEP 2622 H=2622:00 +CPU= 17.627
  01:41:02 STEP 2623 H=2623:00 +CPU= 10.669
  01:41:13 STEP 2624 H=2624:00 +CPU= 11.026
  01:41:30 STEP 2625 H=2625:00 +CPU= 17.154
  01:41:41 STEP 2626 H=2626:00 +CPU= 10.892
  01:41:52 STEP 2627 H=2627:00 +CPU= 10.629
  01:42:11 STEP 2628 H=2628:00 +CPU= 19.010
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002628
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002628
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002628
  01:42:22 STEP 2629 H=2629:00 +CPU= 10.847
  01:42:33 STEP 2630 H=2630:00 +CPU= 11.170
  01:42:50 STEP 2631 H=2631:00 +CPU= 17.205
  01:43:01 STEP 2632 H=2632:00 +CPU= 10.966
  01:43:12 STEP 2633 H=2633:00 +CPU= 10.554
  01:43:29 STEP 2634 H=2634:00 +CPU= 17.130
  01:43:40 STEP 2635 H=2635:00 +CPU= 10.782
  01:43:51 STEP 2636 H=2636:00 +CPU= 11.093
  01:44:08 STEP 2637 H=2637:00 +CPU= 17.543
  01:44:19 STEP 2638 H=2638:00 +CPU= 10.956
  01:44:30 STEP 2639 H=2639:00 +CPU= 10.830
  01:44:51 STEP 2640 H=2640:00 +CPU= 20.784
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002640
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002640
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002640
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002616
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002616
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002616
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002628
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002628
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002628
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_109.zip
Uploading the intermediate file: upload_file_109.zip
  01:45:02 STEP 2641 H=2641:00 +CPU= 10.710
  01:45:13 STEP 2642 H=2642:00 +CPU= 11.201
  01:45:30 STEP 2643 H=2643:00 +CPU= 17.076
  01:45:41 STEP 2644 H=2644:00 +CPU= 10.984
  01:45:52 STEP 2645 H=2645:00 +CPU= 10.801
  01:46:09 STEP 2646 H=2646:00 +CPU= 17.268
  01:46:20 STEP 2647 H=2647:00 +CPU= 10.500
  01:46:31 STEP 2648 H=2648:00 +CPU= 10.981
  01:46:48 STEP 2649 H=2649:00 +CPU= 17.284
  01:46:59 STEP 2650 H=2650:00 +CPU= 10.886
  01:47:10 STEP 2651 H=2651:00 +CPU= 10.817
  01:47:28 STEP 2652 H=2652:00 +CPU= 18.826
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002652
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002652
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002652
  01:47:39 STEP 2653 H=2653:00 +CPU= 10.585
  01:47:50 STEP 2654 H=2654:00 +CPU= 11.043
  01:48:07 STEP 2655 H=2655:00 +CPU= 16.929
  01:48:18 STEP 2656 H=2656:00 +CPU= 10.898
  01:48:29 STEP 2657 H=2657:00 +CPU= 10.691
  01:48:46 STEP 2658 H=2658:00 +CPU= 17.591
  01:48:57 STEP 2659 H=2659:00 +CPU= 10.918
  01:49:08 STEP 2660 H=2660:00 +CPU= 10.737
  01:49:25 STEP 2661 H=2661:00 +CPU= 16.942
  01:49:36 STEP 2662 H=2662:00 +CPU= 11.044
  01:49:47 STEP 2663 H=2663:00 +CPU= 10.861
  01:50:07 STEP 2664 H=2664:00 +CPU= 20.451
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002664
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002664
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002664
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002640
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002640
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002640
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002652
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002652
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002652
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_110.zip
Uploading the intermediate file: upload_file_110.zip
  01:50:18 STEP 2665 H=2665:00 +CPU= 10.790
  01:50:30 STEP 2666 H=2666:00 +CPU= 11.369
  01:50:47 STEP 2667 H=2667:00 +CPU= 17.277
  01:50:58 STEP 2668 H=2668:00 +CPU= 10.790
  01:51:09 STEP 2669 H=2669:00 +CPU= 10.913
  01:51:26 STEP 2670 H=2670:00 +CPU= 17.448
  01:51:37 STEP 2671 H=2671:00 +CPU= 10.700
  01:51:48 STEP 2672 H=2672:00 +CPU= 11.300
  01:52:05 STEP 2673 H=2673:00 +CPU= 16.890
  01:52:16 STEP 2674 H=2674:00 +CPU= 10.983
  01:52:27 STEP 2675 H=2675:00 +CPU= 10.740
  01:52:46 STEP 2676 H=2676:00 +CPU= 18.856
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002676
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002676
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002676
  01:52:56 STEP 2677 H=2677:00 +CPU= 10.652
  01:53:08 STEP 2678 H=2678:00 +CPU= 11.335
  01:53:25 STEP 2679 H=2679:00 +CPU= 17.009
  01:53:36 STEP 2680 H=2680:00 +CPU= 11.065
  01:53:47 STEP 2681 H=2681:00 +CPU= 11.019
  01:54:04 STEP 2682 H=2682:00 +CPU= 17.402
  01:54:15 STEP 2683 H=2683:00 +CPU= 10.850
  01:54:26 STEP 2684 H=2684:00 +CPU= 11.030
  01:54:43 STEP 2685 H=2685:00 +CPU= 17.241
  01:54:54 STEP 2686 H=2686:00 +CPU= 10.696
  01:55:05 STEP 2687 H=2687:00 +CPU= 11.099
  01:55:26 STEP 2688 H=2688:00 +CPU= 20.536
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002688
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002688
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002688
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002664
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002664
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002664
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002676
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002676
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002676
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_111.zip
Uploading the intermediate file: upload_file_111.zip
  01:55:36 STEP 2689 H=2689:00 +CPU= 10.545
  01:55:48 STEP 2690 H=2690:00 +CPU= 11.227
  01:56:05 STEP 2691 H=2691:00 +CPU= 17.266
  01:56:16 STEP 2692 H=2692:00 +CPU= 10.770
  01:56:26 STEP 2693 H=2693:00 +CPU= 10.668
  01:56:44 STEP 2694 H=2694:00 +CPU= 17.469
  01:56:54 STEP 2695 H=2695:00 +CPU= 10.662
  01:57:06 STEP 2696 H=2696:00 +CPU= 11.063
  01:57:23 STEP 2697 H=2697:00 +CPU= 17.214
  01:57:34 STEP 2698 H=2698:00 +CPU= 10.930
  01:57:44 STEP 2699 H=2699:00 +CPU= 10.488
  01:58:03 STEP 2700 H=2700:00 +CPU= 18.827
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002700
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002700
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002700
  01:58:14 STEP 2701 H=2701:00 +CPU= 10.559
  01:58:25 STEP 2702 H=2702:00 +CPU= 10.897
  01:58:42 STEP 2703 H=2703:00 +CPU= 17.449
  01:58:53 STEP 2704 H=2704:00 +CPU= 10.880
  01:59:04 STEP 2705 H=2705:00 +CPU= 10.745
  01:59:21 STEP 2706 H=2706:00 +CPU= 17.680
  01:59:32 STEP 2707 H=2707:00 +CPU= 10.908
  01:59:43 STEP 2708 H=2708:00 +CPU= 10.987
  02:00:00 STEP 2709 H=2709:00 +CPU= 17.156
  02:00:11 STEP 2710 H=2710:00 +CPU= 10.879
  02:00:22 STEP 2711 H=2711:00 +CPU= 11.031
  02:00:43 STEP 2712 H=2712:00 +CPU= 20.420
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002712
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002712
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002712
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002688
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002688
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002688
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002700
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002700
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002700
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_112.zip
Uploading the intermediate file: upload_file_112.zip
  02:00:53 STEP 2713 H=2713:00 +CPU= 10.568
  02:01:05 STEP 2714 H=2714:00 +CPU= 11.388
  02:01:22 STEP 2715 H=2715:00 +CPU= 17.052
  02:01:33 STEP 2716 H=2716:00 +CPU= 11.018
  02:01:44 STEP 2717 H=2717:00 +CPU= 10.662
  02:02:01 STEP 2718 H=2718:00 +CPU= 17.331
  02:02:12 STEP 2719 H=2719:00 +CPU= 10.810
  02:02:23 STEP 2720 H=2720:00 +CPU= 11.227
  02:02:40 STEP 2721 H=2721:00 +CPU= 17.238
  02:02:51 STEP 2722 H=2722:00 +CPU= 10.930
  02:03:02 STEP 2723 H=2723:00 +CPU= 10.830
  02:03:21 STEP 2724 H=2724:00 +CPU= 18.858
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002724
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002724
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002724
  02:03:32 STEP 2725 H=2725:00 +CPU= 10.672
  02:03:43 STEP 2726 H=2726:00 +CPU= 10.945
  02:04:00 STEP 2727 H=2727:00 +CPU= 17.059
  02:04:11 STEP 2728 H=2728:00 +CPU= 11.080
  02:04:22 STEP 2729 H=2729:00 +CPU= 10.903
  02:04:39 STEP 2730 H=2730:00 +CPU= 17.205
  02:04:50 STEP 2731 H=2731:00 +CPU= 10.684
  02:05:00 STEP 2732 H=2732:00 +CPU= 10.989
  02:05:18 STEP 2733 H=2733:00 +CPU= 17.167
  02:05:28 STEP 2734 H=2734:00 +CPU= 10.784
  02:05:39 STEP 2735 H=2735:00 +CPU= 10.967
  02:06:00 STEP 2736 H=2736:00 +CPU= 20.601
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002736
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002736
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002736
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002712
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002712
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002712
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002724
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002724
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002724
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_113.zip
Uploading the intermediate file: upload_file_113.zip
  02:06:11 STEP 2737 H=2737:00 +CPU= 10.912
  02:06:22 STEP 2738 H=2738:00 +CPU= 11.190
  02:06:39 STEP 2739 H=2739:00 +CPU= 17.048
  02:06:50 STEP 2740 H=2740:00 +CPU= 11.034
  02:07:01 STEP 2741 H=2741:00 +CPU= 11.050
  02:07:19 STEP 2742 H=2742:00 +CPU= 17.565
  02:07:30 STEP 2743 H=2743:00 +CPU= 10.531
  02:07:41 STEP 2744 H=2744:00 +CPU= 10.980
  02:07:58 STEP 2745 H=2745:00 +CPU= 17.203
  02:08:09 STEP 2746 H=2746:00 +CPU= 11.200
  02:08:20 STEP 2747 H=2747:00 +CPU= 10.825
  02:08:39 STEP 2748 H=2748:00 +CPU= 19.039
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002748
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002748
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002748
  02:08:50 STEP 2749 H=2749:00 +CPU= 10.727
  02:09:01 STEP 2750 H=2750:00 +CPU= 10.974
  02:09:18 STEP 2751 H=2751:00 +CPU= 17.232
  02:09:29 STEP 2752 H=2752:00 +CPU= 11.073
  02:09:39 STEP 2753 H=2753:00 +CPU= 10.565
  02:09:57 STEP 2754 H=2754:00 +CPU= 17.677
  02:10:08 STEP 2755 H=2755:00 +CPU= 10.784
  02:10:19 STEP 2756 H=2756:00 +CPU= 10.693
  02:10:36 STEP 2757 H=2757:00 +CPU= 17.142
  02:10:47 STEP 2758 H=2758:00 +CPU= 10.714
  02:10:58 STEP 2759 H=2759:00 +CPU= 11.050
  02:11:18 STEP 2760 H=2760:00 +CPU= 20.690
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002760
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002760
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002760
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002736
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002736
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002736
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002748
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002748
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002748
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_114.zip
Uploading the intermediate file: upload_file_114.zip
  02:11:29 STEP 2761 H=2761:00 +CPU= 10.819
Uploading trickle at timestep: 9936000
  02:11:40 STEP 2762 H=2762:00 +CPU= 11.195
  02:11:57 STEP 2763 H=2763:00 +CPU= 16.896
  02:12:08 STEP 2764 H=2764:00 +CPU= 11.184
  02:12:19 STEP 2765 H=2765:00 +CPU= 10.567
  02:12:36 STEP 2766 H=2766:00 +CPU= 17.284
  02:12:47 STEP 2767 H=2767:00 +CPU= 10.758
  02:12:58 STEP 2768 H=2768:00 +CPU= 11.255
  02:13:15 STEP 2769 H=2769:00 +CPU= 16.825
  02:13:26 STEP 2770 H=2770:00 +CPU= 11.161
  02:13:37 STEP 2771 H=2771:00 +CPU= 10.609
  02:13:56 STEP 2772 H=2772:00 +CPU= 19.050
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002772
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002772
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002772
  02:14:07 STEP 2773 H=2773:00 +CPU= 10.407
  02:14:18 STEP 2774 H=2774:00 +CPU= 11.127
  02:14:35 STEP 2775 H=2775:00 +CPU= 17.310
  02:14:46 STEP 2776 H=2776:00 +CPU= 11.000
  02:14:57 STEP 2777 H=2777:00 +CPU= 10.674
  02:15:14 STEP 2778 H=2778:00 +CPU= 17.412
  02:15:25 STEP 2779 H=2779:00 +CPU= 10.759
  02:15:36 STEP 2780 H=2780:00 +CPU= 10.908
  02:15:53 STEP 2781 H=2781:00 +CPU= 17.449
  02:16:04 STEP 2782 H=2782:00 +CPU= 10.801
  02:16:15 STEP 2783 H=2783:00 +CPU= 10.791
  02:16:35 STEP 2784 H=2784:00 +CPU= 20.386
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002784
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002784
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002784
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002760
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002760
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002760
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002772
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002772
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002772
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_115.zip
Uploading the intermediate file: upload_file_115.zip
  02:16:46 STEP 2785 H=2785:00 +CPU= 10.465
  02:16:57 STEP 2786 H=2786:00 +CPU= 11.149
  02:17:14 STEP 2787 H=2787:00 +CPU= 17.027
  02:17:25 STEP 2788 H=2788:00 +CPU= 10.978
  02:17:36 STEP 2789 H=2789:00 +CPU= 10.727
  02:17:54 STEP 2790 H=2790:00 +CPU= 18.043
  02:18:04 STEP 2791 H=2791:00 +CPU= 10.538
  02:18:15 STEP 2792 H=2792:00 +CPU= 11.037
  02:18:33 STEP 2793 H=2793:00 +CPU= 17.484
  02:18:44 STEP 2794 H=2794:00 +CPU= 11.202
  02:18:55 STEP 2795 H=2795:00 +CPU= 10.749
  02:19:14 STEP 2796 H=2796:00 +CPU= 19.217
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002796
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002796
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002796
  02:19:25 STEP 2797 H=2797:00 +CPU= 10.723
  02:19:36 STEP 2798 H=2798:00 +CPU= 10.853
  02:19:53 STEP 2799 H=2799:00 +CPU= 17.217
  02:20:04 STEP 2800 H=2800:00 +CPU= 11.154
  02:20:15 STEP 2801 H=2801:00 +CPU= 10.810
  02:20:33 STEP 2802 H=2802:00 +CPU= 17.646
  02:20:43 STEP 2803 H=2803:00 +CPU= 10.668
  02:20:54 STEP 2804 H=2804:00 +CPU= 10.951
  02:21:11 STEP 2805 H=2805:00 +CPU= 16.973
  02:21:22 STEP 2806 H=2806:00 +CPU= 11.017
  02:21:33 STEP 2807 H=2807:00 +CPU= 10.910
  02:21:54 STEP 2808 H=2808:00 +CPU= 20.418
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002808
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002808
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002808
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002784
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002784
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002784
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002796
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002796
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002796
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_116.zip
Uploading the intermediate file: upload_file_116.zip
  02:22:04 STEP 2809 H=2809:00 +CPU= 10.853
  02:22:15 STEP 2810 H=2810:00 +CPU= 10.979
  02:22:33 STEP 2811 H=2811:00 +CPU= 17.358
  02:22:44 STEP 2812 H=2812:00 +CPU= 10.893
  02:22:54 STEP 2813 H=2813:00 +CPU= 10.497
  02:23:12 STEP 2814 H=2814:00 +CPU= 17.327
  02:23:23 STEP 2815 H=2815:00 +CPU= 10.921
  02:23:33 STEP 2816 H=2816:00 +CPU= 10.845
  02:23:51 STEP 2817 H=2817:00 +CPU= 17.302
  02:24:02 STEP 2818 H=2818:00 +CPU= 11.042
  02:24:13 STEP 2819 H=2819:00 +CPU= 10.955
  02:24:32 STEP 2820 H=2820:00 +CPU= 19.081
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002820
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002820
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002820
  02:24:43 STEP 2821 H=2821:00 +CPU= 10.861
  02:24:54 STEP 2822 H=2822:00 +CPU= 11.133
  02:25:11 STEP 2823 H=2823:00 +CPU= 17.104
  02:25:22 STEP 2824 H=2824:00 +CPU= 11.094
  02:25:33 STEP 2825 H=2825:00 +CPU= 10.561
  02:25:50 STEP 2826 H=2826:00 +CPU= 17.647
  02:26:01 STEP 2827 H=2827:00 +CPU= 10.856
  02:26:12 STEP 2828 H=2828:00 +CPU= 10.826
  02:26:29 STEP 2829 H=2829:00 +CPU= 17.484
  02:26:40 STEP 2830 H=2830:00 +CPU= 10.980
  02:26:51 STEP 2831 H=2831:00 +CPU= 10.963
  02:27:12 STEP 2832 H=2832:00 +CPU= 20.577
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002832
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002832
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002832
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002808
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002808
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002808
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002820
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002820
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002820
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_117.zip
  02:27:23 STEP 2833 H=2833:00 +CPU= 10.547
Uploading the intermediate file: upload_file_117.zip
  02:27:34 STEP 2834 H=2834:00 +CPU= 11.296
  02:27:51 STEP 2835 H=2835:00 +CPU= 17.122
  02:28:02 STEP 2836 H=2836:00 +CPU= 11.050
  02:28:13 STEP 2837 H=2837:00 +CPU= 10.801
  02:28:30 STEP 2838 H=2838:00 +CPU= 17.377
  02:28:41 STEP 2839 H=2839:00 +CPU= 10.594
  02:28:52 STEP 2840 H=2840:00 +CPU= 10.843
  02:29:09 STEP 2841 H=2841:00 +CPU= 16.869
  02:29:20 STEP 2842 H=2842:00 +CPU= 11.011
  02:29:30 STEP 2843 H=2843:00 +CPU= 10.718
  02:29:50 STEP 2844 H=2844:00 +CPU= 19.165
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002844
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002844
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002844
  02:30:00 STEP 2845 H=2845:00 +CPU= 10.784
  02:30:11 STEP 2846 H=2846:00 +CPU= 11.004
  02:30:28 STEP 2847 H=2847:00 +CPU= 16.882
  02:30:39 STEP 2848 H=2848:00 +CPU= 11.145
  02:30:50 STEP 2849 H=2849:00 +CPU= 10.564
  02:31:08 STEP 2850 H=2850:00 +CPU= 17.595
  02:31:18 STEP 2851 H=2851:00 +CPU= 10.763
  02:31:29 STEP 2852 H=2852:00 +CPU= 10.865
  02:31:46 STEP 2853 H=2853:00 +CPU= 17.012
  02:31:57 STEP 2854 H=2854:00 +CPU= 11.006
  02:32:08 STEP 2855 H=2855:00 +CPU= 10.808
  02:32:29 STEP 2856 H=2856:00 +CPU= 20.719
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002856
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002856
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002856
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002832
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002832
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002832
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002844
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002844
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002844
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:32:40 STEP 2857 H=2857:00 +CPU= 10.894
  02:32:51 STEP 2858 H=2858:00 +CPU= 11.029
  02:33:08 STEP 2859 H=2859:00 +CPU= 17.216
  02:33:19 STEP 2860 H=2860:00 +CPU= 10.901
  02:33:30 STEP 2861 H=2861:00 +CPU= 10.831
  02:33:47 STEP 2862 H=2862:00 +CPU= 17.403
  02:33:58 STEP 2863 H=2863:00 +CPU= 10.824
  02:34:09 STEP 2864 H=2864:00 +CPU= 11.029
  02:34:26 STEP 2865 H=2865:00 +CPU= 17.435
  02:34:38 STEP 2866 H=2866:00 +CPU= 11.064
  02:34:48 STEP 2867 H=2867:00 +CPU= 10.821
  02:35:08 STEP 2868 H=2868:00 +CPU= 19.175
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002868
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002868
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002868
  02:35:18 STEP 2869 H=2869:00 +CPU= 10.786
  02:35:30 STEP 2870 H=2870:00 +CPU= 11.276
  02:35:47 STEP 2871 H=2871:00 +CPU= 17.157
  02:35:58 STEP 2872 H=2872:00 +CPU= 11.057
  02:36:09 STEP 2873 H=2873:00 +CPU= 10.902
  02:36:26 STEP 2874 H=2874:00 +CPU= 17.255
  02:36:37 STEP 2875 H=2875:00 +CPU= 10.800
  02:36:48 STEP 2876 H=2876:00 +CPU= 11.134
  02:37:05 STEP 2877 H=2877:00 +CPU= 16.923
  02:37:16 STEP 2878 H=2878:00 +CPU= 10.820
  02:37:27 STEP 2879 H=2879:00 +CPU= 11.056
  02:37:47 STEP 2880 H=2880:00 +CPU= 20.536
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002880
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002880
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002880
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002856
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002856
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002856
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002868
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002868
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002868
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:37:58 STEP 2881 H=2881:00 +CPU= 10.891
  02:38:10 STEP 2882 H=2882:00 +CPU= 11.288
  02:38:27 STEP 2883 H=2883:00 +CPU= 17.047
  02:38:38 STEP 2884 H=2884:00 +CPU= 10.921
  02:38:48 STEP 2885 H=2885:00 +CPU= 10.731
  02:39:06 STEP 2886 H=2886:00 +CPU= 17.489
  02:39:17 STEP 2887 H=2887:00 +CPU= 10.747
  02:39:28 STEP 2888 H=2888:00 +CPU= 11.043
  02:39:45 STEP 2889 H=2889:00 +CPU= 17.210
  02:39:56 STEP 2890 H=2890:00 +CPU= 10.847
  02:40:07 STEP 2891 H=2891:00 +CPU= 10.771
  02:40:26 STEP 2892 H=2892:00 +CPU= 19.092
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002892
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002892
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002892
  02:40:36 STEP 2893 H=2893:00 +CPU= 10.571
  02:40:47 STEP 2894 H=2894:00 +CPU= 11.017
  02:41:04 STEP 2895 H=2895:00 +CPU= 17.101
  02:41:15 STEP 2896 H=2896:00 +CPU= 10.916
  02:41:26 STEP 2897 H=2897:00 +CPU= 10.644
  02:41:43 STEP 2898 H=2898:00 +CPU= 17.466
  02:41:54 STEP 2899 H=2899:00 +CPU= 10.811
  02:42:05 STEP 2900 H=2900:00 +CPU= 10.903
  02:42:22 STEP 2901 H=2901:00 +CPU= 17.104
  02:42:33 STEP 2902 H=2902:00 +CPU= 11.056
  02:42:45 STEP 2903 H=2903:00 +CPU= 11.202
  02:43:05 STEP 2904 H=2904:00 +CPU= 20.548
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002904
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002904
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002904
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002880
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002880
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002880
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002892
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002892
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002892
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_120.zip
Uploading the intermediate file: upload_file_120.zip
  02:43:16 STEP 2905 H=2905:00 +CPU= 10.772
  02:43:27 STEP 2906 H=2906:00 +CPU= 11.232
  02:43:44 STEP 2907 H=2907:00 +CPU= 17.102
  02:43:55 STEP 2908 H=2908:00 +CPU= 10.794
  02:44:06 STEP 2909 H=2909:00 +CPU= 10.784
  02:44:24 STEP 2910 H=2910:00 +CPU= 17.879
  02:44:35 STEP 2911 H=2911:00 +CPU= 10.796
  02:44:46 STEP 2912 H=2912:00 +CPU= 11.133
  02:45:03 STEP 2913 H=2913:00 +CPU= 17.342
  02:45:14 STEP 2914 H=2914:00 +CPU= 11.179
  02:45:25 STEP 2915 H=2915:00 +CPU= 10.682
  02:45:44 STEP 2916 H=2916:00 +CPU= 18.969
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002916
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002916
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002916
  02:45:55 STEP 2917 H=2917:00 +CPU= 10.633
  02:46:06 STEP 2918 H=2918:00 +CPU= 11.009
  02:46:23 STEP 2919 H=2919:00 +CPU= 17.162
  02:46:34 STEP 2920 H=2920:00 +CPU= 10.943
  02:46:45 STEP 2921 H=2921:00 +CPU= 10.963
  02:47:02 STEP 2922 H=2922:00 +CPU= 17.501
  02:47:13 STEP 2923 H=2923:00 +CPU= 10.828
  02:47:24 STEP 2924 H=2924:00 +CPU= 10.888
  02:47:41 STEP 2925 H=2925:00 +CPU= 16.972
  02:47:52 STEP 2926 H=2926:00 +CPU= 11.009
  02:48:03 STEP 2927 H=2927:00 +CPU= 11.388
  02:48:24 STEP 2928 H=2928:00 +CPU= 20.465
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002928
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002928
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002928
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002904
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002904
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002904
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002916
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002916
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002916
Zipping up the intermediate file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_121.zip
Uploading the intermediate file: upload_file_121.zip
  02:48:35 STEP 2929 H=2929:00 +CPU= 10.790
  02:48:46 STEP 2930 H=2930:00 +CPU= 11.227
  02:49:03 STEP 2931 H=2931:00 +CPU= 17.170
  02:49:14 STEP 2932 H=2932:00 +CPU= 11.071
  02:49:25 STEP 2933 H=2933:00 +CPU= 10.703
  02:49:42 STEP 2934 H=2934:00 +CPU= 17.533
  02:49:53 STEP 2935 H=2935:00 +CPU= 10.758
  02:50:04 STEP 2936 H=2936:00 +CPU= 10.864
  02:50:21 STEP 2937 H=2937:00 +CPU= 17.113
  02:50:32 STEP 2938 H=2938:00 +CPU= 11.077
  02:50:43 STEP 2939 H=2939:00 +CPU= 10.721
  02:51:02 STEP 2940 H=2940:00 +CPU= 18.941
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002940
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002940
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002940
  02:51:13 STEP 2941 H=2941:00 +CPU= 10.760
  02:51:24 STEP 2942 H=2942:00 +CPU= 11.167
  02:51:41 STEP 2943 H=2943:00 +CPU= 17.127
  02:51:52 STEP 2944 H=2944:00 +CPU= 10.927
  02:52:03 STEP 2945 H=2945:00 +CPU= 10.522
  02:52:20 STEP 2946 H=2946:00 +CPU= 17.288
  02:52:31 STEP 2947 H=2947:00 +CPU= 10.805
  02:52:42 STEP 2948 H=2948:00 +CPU= 10.869
  02:52:59 STEP 2949 H=2949:00 +CPU= 17.343
  02:53:10 STEP 2950 H=2950:00 +CPU= 11.059
  02:53:21 STEP 2951 H=2951:00 +CPU= 10.955
  02:53:41 STEP 2952 H=2952:00 +CPU= 19.873
The child process terminated with status: 0
Moving to projects directory: /home/ellis/slots/1/ICMGGhq0f+002952
Moving to projects directory: /home/ellis/slots/1/ICMSHhq0f+002952
Moving to projects directory: /home/ellis/slots/1/ICMUAhq0f+002952
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002952
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002952
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002928
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002940
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMSHhq0f+002928
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002928
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002940
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMUAhq0f+002952
Adding to the zip: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_12197327/ICMGGhq0f+002940
Zipping up the final file: /home/ellis/projects/climateprediction.net/oifs_43r3_ps_0683_2011050100_123_980_12197327_0_r136217444_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
02:57:04 (106864): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:28:07 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 10,623,600 38,995 0.0037
09 May 2023 16:28:07 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 9,936,000 36,455 0.0037
09 May 2023 16:28:02 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 9,072,000 33,289 0.0037
09 May 2023 16:27:56 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 8,208,000 30,112 0.0037
09 May 2023 16:27:50 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 7,344,000 26,936 0.0037
09 May 2023 16:27:44 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 6,480,000 23,754 0.0037
09 May 2023 16:27:39 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 5,184,000 19,000 0.0037
09 May 2023 16:27:34 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 4,320,000 15,826 0.0037
09 May 2023 16:27:29 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 3,456,000 12,637 0.0037
09 May 2023 16:27:24 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 2,592,000 9,523 0.0037
09 May 2023 16:27:19 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 1,728,000 6,377 0.0037
09 May 2023 16:27:14 1535413 22283359 oifs_43r3_ps_0683_2011050100_123_980_12197327_0 864,000 3,201 0.0037


©2024 climateprediction.net