climateprediction.net home page
Task 22284699

Task 22284699

Name oifs_43r3_ps_0944_2012050100_123_981_12198588_0
Workunit 12198588
Created 24 Dec 2022, 6:16:54 UTC
Sent 30 Dec 2022, 23:35:03 UTC
Report deadline 29 Jan 2023, 23:35:03 UTC
Received 5 Jan 2023, 0:08:09 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 10 hours 58 min 49 sec
CPU time 10 hours 45 min 47 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,456.36 MB
Peak swap size 4,974.25 MB
Peak disk usage 2,961.45 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
8/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_94.zip
Uploading the intermediate file: upload_file_94.zip
  19:30:53 STEP 2281 H=2281:00 +CPU= 10.488
  19:31:04 STEP 2282 H=2282:00 +CPU= 10.797
  19:31:21 STEP 2283 H=2283:00 +CPU= 17.383
  19:31:33 STEP 2284 H=2284:00 +CPU= 11.442
  19:31:43 STEP 2285 H=2285:00 +CPU= 10.714
  19:32:01 STEP 2286 H=2286:00 +CPU= 17.538
  19:32:12 STEP 2287 H=2287:00 +CPU= 11.513
  19:32:23 STEP 2288 H=2288:00 +CPU= 10.956
  19:32:43 STEP 2289 H=2289:00 +CPU= 17.358
  19:32:53 STEP 2290 H=2290:00 +CPU= 10.714
  19:33:04 STEP 2291 H=2291:00 +CPU= 10.578
  19:33:24 STEP 2292 H=2292:00 +CPU= 19.605
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002292
  19:33:34 STEP 2293 H=2293:00 +CPU= 10.864
  19:33:46 STEP 2294 H=2294:00 +CPU= 11.196
  19:34:03 STEP 2295 H=2295:00 +CPU= 17.229
  19:34:14 STEP 2296 H=2296:00 +CPU= 11.200
  19:34:25 STEP 2297 H=2297:00 +CPU= 11.001
  19:34:43 STEP 2298 H=2298:00 +CPU= 18.142
  19:34:54 STEP 2299 H=2299:00 +CPU= 10.849
  19:35:05 STEP 2300 H=2300:00 +CPU= 10.850
  19:35:23 STEP 2301 H=2301:00 +CPU= 17.656
  19:35:34 STEP 2302 H=2302:00 +CPU= 11.433
  19:35:46 STEP 2303 H=2303:00 +CPU= 11.634
  19:36:08 STEP 2304 H=2304:00 +CPU= 20.036
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_95.zip
Uploading the intermediate file: upload_file_95.zip
  19:36:19 STEP 2305 H=2305:00 +CPU= 10.666
  19:36:30 STEP 2306 H=2306:00 +CPU= 10.815
  19:36:47 STEP 2307 H=2307:00 +CPU= 17.274
  19:36:58 STEP 2308 H=2308:00 +CPU= 11.374
  19:37:09 STEP 2309 H=2309:00 +CPU= 10.715
  19:37:27 STEP 2310 H=2310:00 +CPU= 17.903
  19:37:38 STEP 2311 H=2311:00 +CPU= 10.835
  19:37:49 STEP 2312 H=2312:00 +CPU= 11.113
  19:38:06 STEP 2313 H=2313:00 +CPU= 17.092
  19:38:17 STEP 2314 H=2314:00 +CPU= 10.978
  19:38:28 STEP 2315 H=2315:00 +CPU= 10.766
  19:38:47 STEP 2316 H=2316:00 +CPU= 19.159
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002316
  19:38:58 STEP 2317 H=2317:00 +CPU= 10.476
  19:39:09 STEP 2318 H=2318:00 +CPU= 11.580
  19:39:27 STEP 2319 H=2319:00 +CPU= 17.570
  19:39:38 STEP 2320 H=2320:00 +CPU= 10.869
  19:39:48 STEP 2321 H=2321:00 +CPU= 10.501
  19:40:06 STEP 2322 H=2322:00 +CPU= 17.533
  19:40:17 STEP 2323 H=2323:00 +CPU= 10.916
  19:40:28 STEP 2324 H=2324:00 +CPU= 11.232
  19:40:45 STEP 2325 H=2325:00 +CPU= 17.254
  19:40:57 STEP 2326 H=2326:00 +CPU= 11.800
  19:41:07 STEP 2327 H=2327:00 +CPU= 10.622
  19:41:28 STEP 2328 H=2328:00 +CPU= 20.189
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_96.zip
Uploading the intermediate file: upload_file_96.zip
  19:41:39 STEP 2329 H=2329:00 +CPU= 10.635
  19:41:50 STEP 2330 H=2330:00 +CPU= 10.911
  19:42:08 STEP 2331 H=2331:00 +CPU= 17.764
  19:42:19 STEP 2332 H=2332:00 +CPU= 10.901
  19:42:30 STEP 2333 H=2333:00 +CPU= 11.369
  19:42:47 STEP 2334 H=2334:00 +CPU= 17.147
  19:42:58 STEP 2335 H=2335:00 +CPU= 10.658
  19:43:09 STEP 2336 H=2336:00 +CPU= 11.139
  19:43:26 STEP 2337 H=2337:00 +CPU= 17.039
  19:43:37 STEP 2338 H=2338:00 +CPU= 11.449
  19:43:48 STEP 2339 H=2339:00 +CPU= 10.665
  19:44:08 STEP 2340 H=2340:00 +CPU= 19.437
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002340
  19:44:18 STEP 2341 H=2341:00 +CPU= 10.382
  19:44:29 STEP 2342 H=2342:00 +CPU= 10.948
  19:44:46 STEP 2343 H=2343:00 +CPU= 17.345
  19:44:58 STEP 2344 H=2344:00 +CPU= 10.783
  19:45:09 STEP 2345 H=2345:00 +CPU= 10.784
  19:45:26 STEP 2346 H=2346:00 +CPU= 17.406
  19:45:37 STEP 2347 H=2347:00 +CPU= 10.920
  19:45:48 STEP 2348 H=2348:00 +CPU= 10.755
  19:46:05 STEP 2349 H=2349:00 +CPU= 17.232
  19:46:16 STEP 2350 H=2350:00 +CPU= 10.735
  19:46:27 STEP 2351 H=2351:00 +CPU= 10.718
  19:46:48 STEP 2352 H=2352:00 +CPU= 20.526
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_97.zip
Uploading the intermediate file: upload_file_97.zip
  19:46:58 STEP 2353 H=2353:00 +CPU= 10.560
  19:47:09 STEP 2354 H=2354:00 +CPU= 11.004
  19:47:27 STEP 2355 H=2355:00 +CPU= 17.460
  19:47:38 STEP 2356 H=2356:00 +CPU= 11.041
  19:47:51 STEP 2357 H=2357:00 +CPU= 10.632
  19:48:09 STEP 2358 H=2358:00 +CPU= 17.550
  19:48:19 STEP 2359 H=2359:00 +CPU= 10.795
  19:48:52 STEP 2360 H=2360:00 +CPU= 12.110
  19:49:10 STEP 2361 H=2361:00 +CPU= 17.422
  19:49:21 STEP 2362 H=2362:00 +CPU= 11.033
  19:49:31 STEP 2363 H=2363:00 +CPU= 10.587
  19:49:50 STEP 2364 H=2364:00 +CPU= 18.875
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002364
  19:50:01 STEP 2365 H=2365:00 +CPU= 10.975
  19:50:12 STEP 2366 H=2366:00 +CPU= 11.057
  19:50:28 STEP 2367 H=2367:00 +CPU= 16.220
  19:50:39 STEP 2368 H=2368:00 +CPU= 10.285
  19:50:49 STEP 2369 H=2369:00 +CPU= 10.033
  19:51:05 STEP 2370 H=2370:00 +CPU= 16.374
  19:51:15 STEP 2371 H=2371:00 +CPU=  9.961
  19:51:25 STEP 2372 H=2372:00 +CPU= 10.337
  19:51:42 STEP 2373 H=2373:00 +CPU= 16.998
  19:51:53 STEP 2374 H=2374:00 +CPU= 10.508
  19:52:03 STEP 2375 H=2375:00 +CPU= 10.354
  19:52:23 STEP 2376 H=2376:00 +CPU= 19.391
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_98.zip
Uploading the intermediate file: upload_file_98.zip
  19:52:33 STEP 2377 H=2377:00 +CPU= 10.264
  19:52:44 STEP 2378 H=2378:00 +CPU= 10.328
  19:53:00 STEP 2379 H=2379:00 +CPU= 16.759
  19:53:11 STEP 2380 H=2380:00 +CPU= 10.803
  19:53:22 STEP 2381 H=2381:00 +CPU= 10.503
  19:53:38 STEP 2382 H=2382:00 +CPU= 16.639
  19:53:49 STEP 2383 H=2383:00 +CPU= 10.278
  19:53:59 STEP 2384 H=2384:00 +CPU= 10.495
  19:54:17 STEP 2385 H=2385:00 +CPU= 17.482
  19:54:27 STEP 2386 H=2386:00 +CPU= 10.680
  19:54:38 STEP 2387 H=2387:00 +CPU= 10.412
  19:54:56 STEP 2388 H=2388:00 +CPU= 18.773
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002388
  19:55:07 STEP 2389 H=2389:00 +CPU= 10.373
  19:55:18 STEP 2390 H=2390:00 +CPU= 11.174
  19:55:35 STEP 2391 H=2391:00 +CPU= 16.813
  19:55:45 STEP 2392 H=2392:00 +CPU= 10.464
  19:55:56 STEP 2393 H=2393:00 +CPU= 10.609
  19:56:13 STEP 2394 H=2394:00 +CPU= 17.406
  19:56:24 STEP 2395 H=2395:00 +CPU= 10.524
  19:56:34 STEP 2396 H=2396:00 +CPU= 10.610
  19:56:52 STEP 2397 H=2397:00 +CPU= 17.178
  19:57:03 STEP 2398 H=2398:00 +CPU= 10.824
  19:57:13 STEP 2399 H=2399:00 +CPU= 10.979
  19:57:34 STEP 2400 H=2400:00 +CPU= 20.163
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_99.zip
Uploading the intermediate file: upload_file_99.zip
  19:57:44 STEP 2401 H=2401:00 +CPU= 10.160
  19:57:55 STEP 2402 H=2402:00 +CPU= 10.804
Uploading trickle at timestep: 8640000
  19:58:12 STEP 2403 H=2403:00 +CPU= 16.601
  19:58:22 STEP 2404 H=2404:00 +CPU= 10.809
  19:58:33 STEP 2405 H=2405:00 +CPU= 10.296
  19:58:50 STEP 2406 H=2406:00 +CPU= 17.154
  19:59:00 STEP 2407 H=2407:00 +CPU= 10.422
  19:59:11 STEP 2408 H=2408:00 +CPU= 10.836
  19:59:28 STEP 2409 H=2409:00 +CPU= 17.202
  19:59:39 STEP 2410 H=2410:00 +CPU= 10.543
  19:59:50 STEP 2411 H=2411:00 +CPU= 10.590
  20:00:08 STEP 2412 H=2412:00 +CPU= 18.420
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002412
  20:00:18 STEP 2413 H=2413:00 +CPU= 10.175
  20:00:29 STEP 2414 H=2414:00 +CPU= 10.688
  20:00:46 STEP 2415 H=2415:00 +CPU= 16.886
  20:00:57 STEP 2416 H=2416:00 +CPU= 11.014
  20:01:07 STEP 2417 H=2417:00 +CPU= 10.539
  20:01:24 STEP 2418 H=2418:00 +CPU= 17.099
  20:01:35 STEP 2419 H=2419:00 +CPU= 10.441
  20:01:45 STEP 2420 H=2420:00 +CPU= 10.584
  20:02:02 STEP 2421 H=2421:00 +CPU= 16.718
  20:02:13 STEP 2422 H=2422:00 +CPU= 10.611
  20:02:24 STEP 2423 H=2423:00 +CPU= 11.220
  20:02:44 STEP 2424 H=2424:00 +CPU= 19.789
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_100.zip
Uploading the intermediate file: upload_file_100.zip
  20:02:54 STEP 2425 H=2425:00 +CPU= 10.300
  20:03:05 STEP 2426 H=2426:00 +CPU= 10.738
  20:03:22 STEP 2427 H=2427:00 +CPU= 16.667
  20:03:32 STEP 2428 H=2428:00 +CPU= 10.923
  20:03:43 STEP 2429 H=2429:00 +CPU= 10.342
  20:04:00 STEP 2430 H=2430:00 +CPU= 17.085
  20:04:10 STEP 2431 H=2431:00 +CPU= 10.478
  20:04:21 STEP 2432 H=2432:00 +CPU= 10.630
  20:04:38 STEP 2433 H=2433:00 +CPU= 17.214
  20:04:49 STEP 2434 H=2434:00 +CPU= 10.788
  20:04:59 STEP 2435 H=2435:00 +CPU= 10.407
  20:05:18 STEP 2436 H=2436:00 +CPU= 18.950
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002436
  20:05:29 STEP 2437 H=2437:00 +CPU= 10.418
  20:05:40 STEP 2438 H=2438:00 +CPU= 10.817
  20:05:57 STEP 2439 H=2439:00 +CPU= 17.096
  20:06:08 STEP 2440 H=2440:00 +CPU= 11.253
  20:06:19 STEP 2441 H=2441:00 +CPU= 10.578
  20:06:36 STEP 2442 H=2442:00 +CPU= 17.193
  20:06:46 STEP 2443 H=2443:00 +CPU= 10.167
  20:06:57 STEP 2444 H=2444:00 +CPU= 10.691
  20:07:13 STEP 2445 H=2445:00 +CPU= 16.669
  20:07:24 STEP 2446 H=2446:00 +CPU= 10.509
  20:07:34 STEP 2447 H=2447:00 +CPU= 10.556
  20:07:54 STEP 2448 H=2448:00 +CPU= 19.882
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_101.zip
Uploading the intermediate file: upload_file_101.zip
  20:08:05 STEP 2449 H=2449:00 +CPU= 10.229
  20:08:15 STEP 2450 H=2450:00 +CPU= 10.750
  20:08:32 STEP 2451 H=2451:00 +CPU= 16.661
  20:08:43 STEP 2452 H=2452:00 +CPU= 10.929
  20:08:53 STEP 2453 H=2453:00 +CPU= 10.265
  20:09:10 STEP 2454 H=2454:00 +CPU= 17.217
  20:09:21 STEP 2455 H=2455:00 +CPU= 11.018
  20:09:32 STEP 2456 H=2456:00 +CPU= 10.679
  20:09:49 STEP 2457 H=2457:00 +CPU= 16.921
  20:10:00 STEP 2458 H=2458:00 +CPU= 10.513
  20:10:10 STEP 2459 H=2459:00 +CPU= 10.387
  20:10:28 STEP 2460 H=2460:00 +CPU= 18.486
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002460
  20:10:39 STEP 2461 H=2461:00 +CPU= 10.323
  20:10:50 STEP 2462 H=2462:00 +CPU= 10.828
  20:11:06 STEP 2463 H=2463:00 +CPU= 16.542
  20:11:17 STEP 2464 H=2464:00 +CPU= 11.005
  20:11:28 STEP 2465 H=2465:00 +CPU= 10.472
  20:11:45 STEP 2466 H=2466:00 +CPU= 17.196
  20:11:56 STEP 2467 H=2467:00 +CPU= 10.737
  20:12:06 STEP 2468 H=2468:00 +CPU= 10.906
  20:12:24 STEP 2469 H=2469:00 +CPU= 17.480
  20:12:34 STEP 2470 H=2470:00 +CPU= 10.516
  20:12:45 STEP 2471 H=2471:00 +CPU= 10.633
  20:13:05 STEP 2472 H=2472:00 +CPU= 19.788
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_102.zip
Uploading the intermediate file: upload_file_102.zip
  20:13:15 STEP 2473 H=2473:00 +CPU= 10.109
  20:13:26 STEP 2474 H=2474:00 +CPU= 10.926
  20:13:43 STEP 2475 H=2475:00 +CPU= 17.257
  20:13:54 STEP 2476 H=2476:00 +CPU= 10.870
  20:14:04 STEP 2477 H=2477:00 +CPU= 10.197
  20:14:22 STEP 2478 H=2478:00 +CPU= 17.365
  20:14:32 STEP 2479 H=2479:00 +CPU= 10.352
  20:14:43 STEP 2480 H=2480:00 +CPU= 10.730
  20:14:59 STEP 2481 H=2481:00 +CPU= 16.709
  20:15:10 STEP 2482 H=2482:00 +CPU= 10.673
  20:15:21 STEP 2483 H=2483:00 +CPU= 10.845
  20:15:40 STEP 2484 H=2484:00 +CPU= 18.707
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002484
  20:15:50 STEP 2485 H=2485:00 +CPU= 10.121
  20:16:00 STEP 2486 H=2486:00 +CPU= 10.490
  20:16:17 STEP 2487 H=2487:00 +CPU= 16.527
  20:16:28 STEP 2488 H=2488:00 +CPU= 10.821
  20:16:38 STEP 2489 H=2489:00 +CPU= 10.451
  20:16:55 STEP 2490 H=2490:00 +CPU= 17.047
  20:17:06 STEP 2491 H=2491:00 +CPU= 10.466
  20:17:16 STEP 2492 H=2492:00 +CPU= 10.586
  20:17:34 STEP 2493 H=2493:00 +CPU= 17.436
  20:17:44 STEP 2494 H=2494:00 +CPU= 10.590
  20:17:55 STEP 2495 H=2495:00 +CPU= 10.901
  20:18:15 STEP 2496 H=2496:00 +CPU= 19.487
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_103.zip
Uploading the intermediate file: upload_file_103.zip
  20:18:25 STEP 2497 H=2497:00 +CPU= 10.297
  20:18:36 STEP 2498 H=2498:00 +CPU= 10.560
  20:18:52 STEP 2499 H=2499:00 +CPU= 16.989
  20:19:03 STEP 2500 H=2500:00 +CPU= 10.864
  20:19:14 STEP 2501 H=2501:00 +CPU= 10.524
  20:19:31 STEP 2502 H=2502:00 +CPU= 17.290
  20:19:41 STEP 2503 H=2503:00 +CPU= 10.249
  20:19:52 STEP 2504 H=2504:00 +CPU= 10.688
  20:20:09 STEP 2505 H=2505:00 +CPU= 16.754
  20:20:20 STEP 2506 H=2506:00 +CPU= 10.883
  20:20:30 STEP 2507 H=2507:00 +CPU= 10.576
  20:20:50 STEP 2508 H=2508:00 +CPU= 19.742
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002508
  20:21:01 STEP 2509 H=2509:00 +CPU= 10.425
  20:21:11 STEP 2510 H=2510:00 +CPU= 10.514
  20:21:28 STEP 2511 H=2511:00 +CPU= 16.768
  20:21:39 STEP 2512 H=2512:00 +CPU= 10.975
  20:21:49 STEP 2513 H=2513:00 +CPU= 10.515
  20:22:06 STEP 2514 H=2514:00 +CPU= 17.089
  20:22:17 STEP 2515 H=2515:00 +CPU= 10.516
  20:22:28 STEP 2516 H=2516:00 +CPU= 10.661
  20:22:44 STEP 2517 H=2517:00 +CPU= 16.629
  20:22:55 STEP 2518 H=2518:00 +CPU= 10.784
  20:23:06 STEP 2519 H=2519:00 +CPU= 10.975
  20:23:26 STEP 2520 H=2520:00 +CPU= 20.301
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_104.zip
Uploading the intermediate file: upload_file_104.zip
  20:23:37 STEP 2521 H=2521:00 +CPU= 10.334
  20:23:47 STEP 2522 H=2522:00 +CPU= 10.493
  20:24:04 STEP 2523 H=2523:00 +CPU= 16.831
  20:24:15 STEP 2524 H=2524:00 +CPU= 10.615
  20:24:25 STEP 2525 H=2525:00 +CPU= 10.433
  20:24:42 STEP 2526 H=2526:00 +CPU= 16.665
  20:24:52 STEP 2527 H=2527:00 +CPU= 10.527
  20:25:03 STEP 2528 H=2528:00 +CPU= 10.811
  20:25:20 STEP 2529 H=2529:00 +CPU= 17.319
  20:25:31 STEP 2530 H=2530:00 +CPU= 10.597
  20:25:42 STEP 2531 H=2531:00 +CPU= 10.535
  20:26:01 STEP 2532 H=2532:00 +CPU= 19.229
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002532
  20:26:11 STEP 2533 H=2533:00 +CPU= 10.355
  20:26:22 STEP 2534 H=2534:00 +CPU= 10.963
  20:26:39 STEP 2535 H=2535:00 +CPU= 16.891
  20:26:51 STEP 2536 H=2536:00 +CPU= 10.643
  20:27:01 STEP 2537 H=2537:00 +CPU= 10.251
  20:27:18 STEP 2538 H=2538:00 +CPU= 16.983
  20:27:28 STEP 2539 H=2539:00 +CPU= 10.582
  20:27:40 STEP 2540 H=2540:00 +CPU= 11.171
  20:27:57 STEP 2541 H=2541:00 +CPU= 17.398
  20:28:08 STEP 2542 H=2542:00 +CPU= 11.107
  20:28:19 STEP 2543 H=2543:00 +CPU= 10.874
  20:28:39 STEP 2544 H=2544:00 +CPU= 19.691
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_105.zip
Uploading the intermediate file: upload_file_105.zip
  20:28:49 STEP 2545 H=2545:00 +CPU= 10.656
  20:29:00 STEP 2546 H=2546:00 +CPU= 10.778
  20:29:16 STEP 2547 H=2547:00 +CPU= 16.338
  20:29:27 STEP 2548 H=2548:00 +CPU= 10.619
  20:29:38 STEP 2549 H=2549:00 +CPU= 10.515
  20:29:54 STEP 2550 H=2550:00 +CPU= 16.891
  20:30:05 STEP 2551 H=2551:00 +CPU= 10.356
  20:30:15 STEP 2552 H=2552:00 +CPU= 10.579
  20:30:33 STEP 2553 H=2553:00 +CPU= 17.308
  20:30:43 STEP 2554 H=2554:00 +CPU= 10.703
  20:30:54 STEP 2555 H=2555:00 +CPU= 10.328
  20:31:12 STEP 2556 H=2556:00 +CPU= 18.426
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002556
  20:31:23 STEP 2557 H=2557:00 +CPU= 10.922
  20:31:34 STEP 2558 H=2558:00 +CPU= 10.756
  20:31:51 STEP 2559 H=2559:00 +CPU= 16.770
  20:32:02 STEP 2560 H=2560:00 +CPU= 10.932
  20:32:12 STEP 2561 H=2561:00 +CPU= 10.221
  20:32:29 STEP 2562 H=2562:00 +CPU= 17.221
  20:32:39 STEP 2563 H=2563:00 +CPU= 10.411
  20:32:50 STEP 2564 H=2564:00 +CPU= 10.545
  20:33:07 STEP 2565 H=2565:00 +CPU= 17.241
  20:33:18 STEP 2566 H=2566:00 +CPU= 10.570
  20:33:28 STEP 2567 H=2567:00 +CPU= 10.513
  20:33:51 STEP 2568 H=2568:00 +CPU= 19.993
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:34:02 STEP 2569 H=2569:00 +CPU= 10.291
  20:34:12 STEP 2570 H=2570:00 +CPU= 10.635
  20:34:29 STEP 2571 H=2571:00 +CPU= 16.958
  20:34:40 STEP 2572 H=2572:00 +CPU= 10.975
  20:34:53 STEP 2573 H=2573:00 +CPU=  9.996
  20:35:10 STEP 2574 H=2574:00 +CPU= 17.077
  20:35:21 STEP 2575 H=2575:00 +CPU= 10.565
  20:35:31 STEP 2576 H=2576:00 +CPU= 10.478
  20:35:48 STEP 2577 H=2577:00 +CPU= 17.131
  20:35:59 STEP 2578 H=2578:00 +CPU= 11.243
  20:36:10 STEP 2579 H=2579:00 +CPU= 11.026
  20:36:29 STEP 2580 H=2580:00 +CPU= 18.569
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002580
  20:36:40 STEP 2581 H=2581:00 +CPU= 10.631
  20:36:51 STEP 2582 H=2582:00 +CPU= 10.987
  20:37:08 STEP 2583 H=2583:00 +CPU= 17.162
  20:37:20 STEP 2584 H=2584:00 +CPU= 10.687
  20:37:30 STEP 2585 H=2585:00 +CPU= 10.606
  20:37:48 STEP 2586 H=2586:00 +CPU= 17.181
  20:37:58 STEP 2587 H=2587:00 +CPU= 10.294
  20:38:08 STEP 2588 H=2588:00 +CPU= 10.409
  20:38:25 STEP 2589 H=2589:00 +CPU= 16.455
  20:38:35 STEP 2590 H=2590:00 +CPU= 10.554
  20:38:46 STEP 2591 H=2591:00 +CPU= 10.874
  20:39:06 STEP 2592 H=2592:00 +CPU= 19.428
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:39:17 STEP 2593 H=2593:00 +CPU= 11.263
  20:39:28 STEP 2594 H=2594:00 +CPU= 10.619
  20:39:44 STEP 2595 H=2595:00 +CPU= 16.696
  20:39:55 STEP 2596 H=2596:00 +CPU= 10.596
  20:40:05 STEP 2597 H=2597:00 +CPU= 10.373
  20:40:23 STEP 2598 H=2598:00 +CPU= 17.508
  20:40:33 STEP 2599 H=2599:00 +CPU= 10.216
  20:40:43 STEP 2600 H=2600:00 +CPU= 10.450
  20:41:00 STEP 2601 H=2601:00 +CPU= 16.611
  20:41:10 STEP 2602 H=2602:00 +CPU= 10.471
  20:41:21 STEP 2603 H=2603:00 +CPU= 10.413
  20:41:39 STEP 2604 H=2604:00 +CPU= 18.552
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002604
  20:41:50 STEP 2605 H=2605:00 +CPU= 10.914
  20:42:01 STEP 2606 H=2606:00 +CPU= 10.567
  20:42:18 STEP 2607 H=2607:00 +CPU= 16.852
  20:42:29 STEP 2608 H=2608:00 +CPU= 10.981
  20:42:39 STEP 2609 H=2609:00 +CPU= 10.241
  20:42:57 STEP 2610 H=2610:00 +CPU= 17.644
  20:43:07 STEP 2611 H=2611:00 +CPU= 10.253
  20:43:18 STEP 2612 H=2612:00 +CPU= 10.880
  20:43:34 STEP 2613 H=2613:00 +CPU= 16.671
  20:43:45 STEP 2614 H=2614:00 +CPU= 10.317
  20:43:55 STEP 2615 H=2615:00 +CPU= 10.553
  20:44:15 STEP 2616 H=2616:00 +CPU= 19.516
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_108.zip
Uploading the intermediate file: upload_file_108.zip
  20:44:25 STEP 2617 H=2617:00 +CPU= 10.510
  20:44:36 STEP 2618 H=2618:00 +CPU= 10.565
  20:44:53 STEP 2619 H=2619:00 +CPU= 17.376
  20:45:04 STEP 2620 H=2620:00 +CPU= 10.335
  20:45:14 STEP 2621 H=2621:00 +CPU= 10.479
  20:45:32 STEP 2622 H=2622:00 +CPU= 17.407
  20:45:42 STEP 2623 H=2623:00 +CPU= 10.243
  20:45:52 STEP 2624 H=2624:00 +CPU= 10.373
  20:46:09 STEP 2625 H=2625:00 +CPU= 16.905
  20:46:19 STEP 2626 H=2626:00 +CPU= 10.332
  20:46:30 STEP 2627 H=2627:00 +CPU= 10.357
  20:46:48 STEP 2628 H=2628:00 +CPU= 18.564
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002628
  20:46:59 STEP 2629 H=2629:00 +CPU= 10.658
  20:47:10 STEP 2630 H=2630:00 +CPU= 10.616
  20:47:26 STEP 2631 H=2631:00 +CPU= 16.772
  20:47:37 STEP 2632 H=2632:00 +CPU= 10.362
  20:47:47 STEP 2633 H=2633:00 +CPU= 10.657
  20:48:06 STEP 2634 H=2634:00 +CPU= 18.134
  20:48:16 STEP 2635 H=2635:00 +CPU= 10.025
  20:48:26 STEP 2636 H=2636:00 +CPU= 10.751
  20:48:43 STEP 2637 H=2637:00 +CPU= 16.487
  20:48:53 STEP 2638 H=2638:00 +CPU= 10.589
  20:49:04 STEP 2639 H=2639:00 +CPU= 10.699
  20:49:24 STEP 2640 H=2640:00 +CPU= 19.747
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_109.zip
Uploading the intermediate file: upload_file_109.zip
  20:49:34 STEP 2641 H=2641:00 +CPU= 10.526
  20:49:45 STEP 2642 H=2642:00 +CPU= 10.436
Uploading trickle at timestep: 9504000
  20:50:02 STEP 2643 H=2643:00 +CPU= 16.893
  20:50:12 STEP 2644 H=2644:00 +CPU= 10.633
  20:50:23 STEP 2645 H=2645:00 +CPU= 10.406
  20:50:40 STEP 2646 H=2646:00 +CPU= 17.147
  20:50:50 STEP 2647 H=2647:00 +CPU= 10.231
  20:51:01 STEP 2648 H=2648:00 +CPU= 10.597
  20:51:17 STEP 2649 H=2649:00 +CPU= 16.617
  20:51:28 STEP 2650 H=2650:00 +CPU= 10.458
  20:51:38 STEP 2651 H=2651:00 +CPU= 10.582
  20:51:57 STEP 2652 H=2652:00 +CPU= 18.763
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002652
  20:52:08 STEP 2653 H=2653:00 +CPU= 10.727
  20:52:19 STEP 2654 H=2654:00 +CPU= 10.645
  20:52:35 STEP 2655 H=2655:00 +CPU= 16.883
  20:52:46 STEP 2656 H=2656:00 +CPU= 10.778
  20:52:57 STEP 2657 H=2657:00 +CPU= 10.424
  20:53:14 STEP 2658 H=2658:00 +CPU= 17.248
  20:53:24 STEP 2659 H=2659:00 +CPU= 10.234
  20:53:35 STEP 2660 H=2660:00 +CPU= 10.636
  20:53:52 STEP 2661 H=2661:00 +CPU= 16.903
  20:54:02 STEP 2662 H=2662:00 +CPU= 10.528
  20:54:13 STEP 2663 H=2663:00 +CPU= 10.792
  20:54:33 STEP 2664 H=2664:00 +CPU= 20.084
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_110.zip
Uploading the intermediate file: upload_file_110.zip
  20:54:43 STEP 2665 H=2665:00 +CPU= 10.338
  20:54:54 STEP 2666 H=2666:00 +CPU= 10.598
  20:55:11 STEP 2667 H=2667:00 +CPU= 16.890
  20:55:21 STEP 2668 H=2668:00 +CPU= 10.618
  20:55:32 STEP 2669 H=2669:00 +CPU= 10.532
  20:55:49 STEP 2670 H=2670:00 +CPU= 17.288
  20:56:00 STEP 2671 H=2671:00 +CPU= 10.314
  20:56:11 STEP 2672 H=2672:00 +CPU= 10.875
  20:56:27 STEP 2673 H=2673:00 +CPU= 16.436
  20:56:38 STEP 2674 H=2674:00 +CPU= 10.755
  20:56:48 STEP 2675 H=2675:00 +CPU= 10.631
  20:57:07 STEP 2676 H=2676:00 +CPU= 18.578
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002676
  20:57:18 STEP 2677 H=2677:00 +CPU= 10.800
  20:57:29 STEP 2678 H=2678:00 +CPU= 11.055
  20:57:46 STEP 2679 H=2679:00 +CPU= 16.855
  20:57:57 STEP 2680 H=2680:00 +CPU= 10.903
  20:58:07 STEP 2681 H=2681:00 +CPU= 10.376
  20:58:25 STEP 2682 H=2682:00 +CPU= 17.608
  20:58:35 STEP 2683 H=2683:00 +CPU= 10.287
  20:58:46 STEP 2684 H=2684:00 +CPU= 10.780
  20:59:02 STEP 2685 H=2685:00 +CPU= 16.662
  20:59:13 STEP 2686 H=2686:00 +CPU= 11.090
  20:59:24 STEP 2687 H=2687:00 +CPU= 10.861
  20:59:44 STEP 2688 H=2688:00 +CPU= 19.456
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_111.zip
Uploading the intermediate file: upload_file_111.zip
  20:59:54 STEP 2689 H=2689:00 +CPU= 10.313
  21:00:05 STEP 2690 H=2690:00 +CPU= 10.881
  21:00:22 STEP 2691 H=2691:00 +CPU= 16.856
  21:00:33 STEP 2692 H=2692:00 +CPU= 10.877
  21:00:43 STEP 2693 H=2693:00 +CPU= 10.395
  21:01:01 STEP 2694 H=2694:00 +CPU= 17.538
  21:01:11 STEP 2695 H=2695:00 +CPU= 10.176
  21:01:22 STEP 2696 H=2696:00 +CPU= 10.820
  21:01:38 STEP 2697 H=2697:00 +CPU= 16.683
  21:01:49 STEP 2698 H=2698:00 +CPU= 10.408
  21:01:59 STEP 2699 H=2699:00 +CPU= 10.794
  21:02:18 STEP 2700 H=2700:00 +CPU= 18.983
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002700
  21:02:29 STEP 2701 H=2701:00 +CPU= 10.558
  21:02:40 STEP 2702 H=2702:00 +CPU= 10.547
  21:02:57 STEP 2703 H=2703:00 +CPU= 16.985
  21:03:07 STEP 2704 H=2704:00 +CPU= 10.982
  21:03:18 STEP 2705 H=2705:00 +CPU= 10.332
  21:03:35 STEP 2706 H=2706:00 +CPU= 17.284
  21:03:45 STEP 2707 H=2707:00 +CPU= 10.011
  21:03:56 STEP 2708 H=2708:00 +CPU= 11.069
  21:04:13 STEP 2709 H=2709:00 +CPU= 16.592
  21:04:23 STEP 2710 H=2710:00 +CPU= 10.455
  21:04:34 STEP 2711 H=2711:00 +CPU= 10.555
  21:04:54 STEP 2712 H=2712:00 +CPU= 19.884
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_112.zip
Uploading the intermediate file: upload_file_112.zip
  21:05:04 STEP 2713 H=2713:00 +CPU= 10.721
  21:05:15 STEP 2714 H=2714:00 +CPU= 10.758
  21:05:32 STEP 2715 H=2715:00 +CPU= 16.545
  21:05:42 STEP 2716 H=2716:00 +CPU= 10.484
  21:05:53 STEP 2717 H=2717:00 +CPU= 10.712
  21:06:10 STEP 2718 H=2718:00 +CPU= 16.963
  21:06:20 STEP 2719 H=2719:00 +CPU= 10.261
  21:06:31 STEP 2720 H=2720:00 +CPU= 11.330
  21:06:48 STEP 2721 H=2721:00 +CPU= 16.632
  21:06:59 STEP 2722 H=2722:00 +CPU= 11.037
  21:07:10 STEP 2723 H=2723:00 +CPU= 10.442
  21:07:28 STEP 2724 H=2724:00 +CPU= 18.801
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002724
  21:07:39 STEP 2725 H=2725:00 +CPU= 10.693
  21:07:50 STEP 2726 H=2726:00 +CPU= 10.693
  21:08:07 STEP 2727 H=2727:00 +CPU= 17.050
  21:08:17 STEP 2728 H=2728:00 +CPU= 10.638
  21:08:28 STEP 2729 H=2729:00 +CPU= 10.698
  21:08:45 STEP 2730 H=2730:00 +CPU= 17.147
  21:08:56 STEP 2731 H=2731:00 +CPU= 10.643
  21:09:07 STEP 2732 H=2732:00 +CPU= 10.770
  21:09:23 STEP 2733 H=2733:00 +CPU= 16.634
  21:09:34 STEP 2734 H=2734:00 +CPU= 10.808
  21:09:45 STEP 2735 H=2735:00 +CPU= 10.652
  21:10:05 STEP 2736 H=2736:00 +CPU= 20.009
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_113.zip
Uploading the intermediate file: upload_file_113.zip
  21:10:15 STEP 2737 H=2737:00 +CPU= 10.237
  21:10:26 STEP 2738 H=2738:00 +CPU= 10.665
  21:10:43 STEP 2739 H=2739:00 +CPU= 17.029
  21:10:53 STEP 2740 H=2740:00 +CPU= 10.509
  21:11:04 STEP 2741 H=2741:00 +CPU= 10.807
  21:11:21 STEP 2742 H=2742:00 +CPU= 17.097
  21:11:32 STEP 2743 H=2743:00 +CPU= 10.572
  21:11:43 STEP 2744 H=2744:00 +CPU= 11.463
  21:12:00 STEP 2745 H=2745:00 +CPU= 16.636
  21:12:11 STEP 2746 H=2746:00 +CPU= 10.943
  21:12:21 STEP 2747 H=2747:00 +CPU= 10.324
  21:12:40 STEP 2748 H=2748:00 +CPU= 18.886
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002748
  21:12:51 STEP 2749 H=2749:00 +CPU= 10.347
  21:13:01 STEP 2750 H=2750:00 +CPU= 10.625
  21:13:18 STEP 2751 H=2751:00 +CPU= 16.715
  21:13:28 STEP 2752 H=2752:00 +CPU= 10.370
  21:13:39 STEP 2753 H=2753:00 +CPU= 10.602
  21:13:56 STEP 2754 H=2754:00 +CPU= 17.475
  21:14:07 STEP 2755 H=2755:00 +CPU= 10.553
  21:14:18 STEP 2756 H=2756:00 +CPU= 11.020
  21:14:35 STEP 2757 H=2757:00 +CPU= 16.616
  21:14:45 STEP 2758 H=2758:00 +CPU= 10.807
  21:14:56 STEP 2759 H=2759:00 +CPU= 11.005
  21:15:16 STEP 2760 H=2760:00 +CPU= 19.594
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_114.zip
Uploading the intermediate file: upload_file_114.zip
  21:15:26 STEP 2761 H=2761:00 +CPU= 10.356
  21:15:37 STEP 2762 H=2762:00 +CPU= 10.459
  21:15:53 STEP 2763 H=2763:00 +CPU= 16.599
  21:16:04 STEP 2764 H=2764:00 +CPU= 11.014
  21:16:15 STEP 2765 H=2765:00 +CPU= 10.539
  21:16:32 STEP 2766 H=2766:00 +CPU= 17.114
  21:16:43 STEP 2767 H=2767:00 +CPU= 10.533
  21:16:54 STEP 2768 H=2768:00 +CPU= 10.871
  21:17:11 STEP 2769 H=2769:00 +CPU= 17.094
  21:17:21 STEP 2770 H=2770:00 +CPU= 10.572
  21:17:31 STEP 2771 H=2771:00 +CPU= 10.217
  21:17:50 STEP 2772 H=2772:00 +CPU= 18.581
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002772
  21:18:01 STEP 2773 H=2773:00 +CPU= 10.744
  21:18:12 STEP 2774 H=2774:00 +CPU= 10.864
  21:18:28 STEP 2775 H=2775:00 +CPU= 16.745
  21:18:39 STEP 2776 H=2776:00 +CPU= 10.613
  21:18:49 STEP 2777 H=2777:00 +CPU= 10.183
  21:19:06 STEP 2778 H=2778:00 +CPU= 16.851
  21:19:16 STEP 2779 H=2779:00 +CPU= 10.507
  21:19:27 STEP 2780 H=2780:00 +CPU= 10.789
  21:19:44 STEP 2781 H=2781:00 +CPU= 16.802
  21:19:55 STEP 2782 H=2782:00 +CPU= 10.914
  21:20:06 STEP 2783 H=2783:00 +CPU= 10.777
  21:20:26 STEP 2784 H=2784:00 +CPU= 20.380
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_115.zip
Uploading the intermediate file: upload_file_115.zip
  21:20:36 STEP 2785 H=2785:00 +CPU= 10.090
  21:20:47 STEP 2786 H=2786:00 +CPU= 10.669
  21:21:04 STEP 2787 H=2787:00 +CPU= 16.694
  21:21:15 STEP 2788 H=2788:00 +CPU= 10.960
  21:21:25 STEP 2789 H=2789:00 +CPU= 10.429
  21:21:42 STEP 2790 H=2790:00 +CPU= 16.613
  21:21:52 STEP 2791 H=2791:00 +CPU= 10.479
  21:22:03 STEP 2792 H=2792:00 +CPU= 10.578
  21:22:19 STEP 2793 H=2793:00 +CPU= 16.738
  21:22:30 STEP 2794 H=2794:00 +CPU= 10.765
  21:22:41 STEP 2795 H=2795:00 +CPU= 10.319
  21:23:00 STEP 2796 H=2796:00 +CPU= 19.081
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002796
  21:23:10 STEP 2797 H=2797:00 +CPU= 10.368
  21:23:21 STEP 2798 H=2798:00 +CPU= 10.844
  21:23:38 STEP 2799 H=2799:00 +CPU= 17.030
  21:23:49 STEP 2800 H=2800:00 +CPU= 10.729
  21:23:59 STEP 2801 H=2801:00 +CPU= 10.497
  21:24:16 STEP 2802 H=2802:00 +CPU= 16.955
  21:24:26 STEP 2803 H=2803:00 +CPU= 10.343
  21:24:37 STEP 2804 H=2804:00 +CPU= 10.415
  21:24:53 STEP 2805 H=2805:00 +CPU= 16.531
  21:25:04 STEP 2806 H=2806:00 +CPU= 10.699
  21:25:15 STEP 2807 H=2807:00 +CPU= 10.570
  21:25:34 STEP 2808 H=2808:00 +CPU= 19.608
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_116.zip
Uploading the intermediate file: upload_file_116.zip
  21:25:45 STEP 2809 H=2809:00 +CPU= 10.322
  21:25:55 STEP 2810 H=2810:00 +CPU= 10.712
  21:26:12 STEP 2811 H=2811:00 +CPU= 17.029
  21:26:23 STEP 2812 H=2812:00 +CPU= 10.783
  21:26:34 STEP 2813 H=2813:00 +CPU= 10.574
  21:26:51 STEP 2814 H=2814:00 +CPU= 16.997
  21:27:01 STEP 2815 H=2815:00 +CPU= 10.397
  21:27:12 STEP 2816 H=2816:00 +CPU= 10.663
  21:27:29 STEP 2817 H=2817:00 +CPU= 16.764
  21:27:39 STEP 2818 H=2818:00 +CPU= 10.618
  21:27:49 STEP 2819 H=2819:00 +CPU= 10.238
  21:28:07 STEP 2820 H=2820:00 +CPU= 18.035
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002820
  21:28:18 STEP 2821 H=2821:00 +CPU= 10.348
  21:28:28 STEP 2822 H=2822:00 +CPU= 10.719
  21:28:45 STEP 2823 H=2823:00 +CPU= 16.558
  21:28:56 STEP 2824 H=2824:00 +CPU= 10.620
  21:29:06 STEP 2825 H=2825:00 +CPU= 10.165
  21:29:23 STEP 2826 H=2826:00 +CPU= 16.683
  21:29:33 STEP 2827 H=2827:00 +CPU= 10.548
  21:29:44 STEP 2828 H=2828:00 +CPU= 10.481
  21:29:59 STEP 2829 H=2829:00 +CPU= 15.800
  21:30:09 STEP 2830 H=2830:00 +CPU= 10.085
  21:30:19 STEP 2831 H=2831:00 +CPU= 10.047
  21:30:38 STEP 2832 H=2832:00 +CPU= 18.431
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_117.zip
Uploading the intermediate file: upload_file_117.zip
  21:30:48 STEP 2833 H=2833:00 +CPU=  9.691
  21:30:58 STEP 2834 H=2834:00 +CPU=  9.910
  21:31:14 STEP 2835 H=2835:00 +CPU= 16.749
  21:31:24 STEP 2836 H=2836:00 +CPU=  9.991
  21:31:34 STEP 2837 H=2837:00 +CPU=  9.717
  21:31:50 STEP 2838 H=2838:00 +CPU= 16.079
  21:32:00 STEP 2839 H=2839:00 +CPU=  9.761
  21:32:10 STEP 2840 H=2840:00 +CPU=  9.991
  21:32:26 STEP 2841 H=2841:00 +CPU= 16.123
  21:32:36 STEP 2842 H=2842:00 +CPU= 10.011
  21:32:46 STEP 2843 H=2843:00 +CPU=  9.752
  21:33:03 STEP 2844 H=2844:00 +CPU= 17.569
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002844
  21:33:13 STEP 2845 H=2845:00 +CPU=  9.732
  21:33:23 STEP 2846 H=2846:00 +CPU= 10.106
  21:33:39 STEP 2847 H=2847:00 +CPU= 16.315
  21:33:50 STEP 2848 H=2848:00 +CPU= 10.765
  21:34:00 STEP 2849 H=2849:00 +CPU=  9.710
  21:34:16 STEP 2850 H=2850:00 +CPU= 16.112
  21:34:26 STEP 2851 H=2851:00 +CPU=  9.792
  21:34:36 STEP 2852 H=2852:00 +CPU=  9.985
  21:34:52 STEP 2853 H=2853:00 +CPU= 15.859
  21:35:02 STEP 2854 H=2854:00 +CPU= 10.003
  21:35:12 STEP 2855 H=2855:00 +CPU= 10.024
  21:35:31 STEP 2856 H=2856:00 +CPU= 19.460
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_118.zip
Uploading the intermediate file: upload_file_118.zip
  21:35:42 STEP 2857 H=2857:00 +CPU= 10.457
  21:35:52 STEP 2858 H=2858:00 +CPU=  9.989
  21:36:07 STEP 2859 H=2859:00 +CPU= 15.798
  21:36:18 STEP 2860 H=2860:00 +CPU= 10.377
  21:36:28 STEP 2861 H=2861:00 +CPU=  9.978
  21:36:44 STEP 2862 H=2862:00 +CPU= 16.090
  21:36:54 STEP 2863 H=2863:00 +CPU= 10.007
  21:37:04 STEP 2864 H=2864:00 +CPU= 10.153
  21:37:20 STEP 2865 H=2865:00 +CPU= 16.214
  21:37:31 STEP 2866 H=2866:00 +CPU= 10.386
  21:37:40 STEP 2867 H=2867:00 +CPU=  9.734
  21:37:58 STEP 2868 H=2868:00 +CPU= 17.508
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002868
  21:38:08 STEP 2869 H=2869:00 +CPU=  9.730
  21:38:18 STEP 2870 H=2870:00 +CPU= 10.014
  21:38:33 STEP 2871 H=2871:00 +CPU= 15.664
  21:38:43 STEP 2872 H=2872:00 +CPU= 10.041
  21:38:53 STEP 2873 H=2873:00 +CPU=  9.695
  21:39:11 STEP 2874 H=2874:00 +CPU= 17.570
  21:39:20 STEP 2875 H=2875:00 +CPU=  9.730
  21:39:30 STEP 2876 H=2876:00 +CPU=  9.979
  21:39:46 STEP 2877 H=2877:00 +CPU= 15.770
  21:39:57 STEP 2878 H=2878:00 +CPU= 10.624
  21:40:07 STEP 2879 H=2879:00 +CPU=  9.934
  21:40:25 STEP 2880 H=2880:00 +CPU= 18.625
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_119.zip
Uploading the intermediate file: upload_file_119.zip
  21:40:36 STEP 2881 H=2881:00 +CPU= 10.447
  21:40:46 STEP 2882 H=2882:00 +CPU= 10.167
Uploading trickle at timestep: 10368000
  21:41:02 STEP 2883 H=2883:00 +CPU= 16.375
  21:41:13 STEP 2884 H=2884:00 +CPU= 10.331
  21:41:22 STEP 2885 H=2885:00 +CPU=  9.748
  21:41:39 STEP 2886 H=2886:00 +CPU= 16.392
  21:41:49 STEP 2887 H=2887:00 +CPU=  9.885
  21:41:59 STEP 2888 H=2888:00 +CPU=  9.979
  21:42:15 STEP 2889 H=2889:00 +CPU= 16.404
  21:42:25 STEP 2890 H=2890:00 +CPU= 10.071
  21:42:35 STEP 2891 H=2891:00 +CPU=  9.743
  21:42:53 STEP 2892 H=2892:00 +CPU= 18.262
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002892
  21:43:03 STEP 2893 H=2893:00 +CPU=  9.801
  21:43:13 STEP 2894 H=2894:00 +CPU= 10.350
  21:43:29 STEP 2895 H=2895:00 +CPU= 15.820
  21:43:39 STEP 2896 H=2896:00 +CPU= 10.052
  21:43:49 STEP 2897 H=2897:00 +CPU=  9.708
  21:44:05 STEP 2898 H=2898:00 +CPU= 16.005
  21:44:15 STEP 2899 H=2899:00 +CPU=  9.735
  21:44:24 STEP 2900 H=2900:00 +CPU=  9.920
  21:44:41 STEP 2901 H=2901:00 +CPU= 16.038
  21:44:51 STEP 2902 H=2902:00 +CPU= 10.483
  21:45:01 STEP 2903 H=2903:00 +CPU= 10.008
  21:45:20 STEP 2904 H=2904:00 +CPU= 18.523
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_120.zip
Uploading the intermediate file: upload_file_120.zip
  21:45:29 STEP 2905 H=2905:00 +CPU=  9.755
  21:45:40 STEP 2906 H=2906:00 +CPU= 10.431
  21:45:56 STEP 2907 H=2907:00 +CPU= 15.898
  21:46:06 STEP 2908 H=2908:00 +CPU= 10.323
  21:46:16 STEP 2909 H=2909:00 +CPU=  9.977
  21:46:32 STEP 2910 H=2910:00 +CPU= 16.028
  21:46:42 STEP 2911 H=2911:00 +CPU=  9.723
  21:46:52 STEP 2912 H=2912:00 +CPU=  9.979
  21:47:07 STEP 2913 H=2913:00 +CPU= 15.613
  21:47:17 STEP 2914 H=2914:00 +CPU=  9.982
  21:47:27 STEP 2915 H=2915:00 +CPU=  9.689
  21:47:45 STEP 2916 H=2916:00 +CPU= 18.261
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002916
  21:47:56 STEP 2917 H=2917:00 +CPU= 10.380
  21:48:06 STEP 2918 H=2918:00 +CPU= 10.064
  21:48:22 STEP 2919 H=2919:00 +CPU= 16.245
  21:48:32 STEP 2920 H=2920:00 +CPU= 10.304
  21:48:42 STEP 2921 H=2921:00 +CPU=  9.696
  21:48:59 STEP 2922 H=2922:00 +CPU= 16.687
  21:49:08 STEP 2923 H=2923:00 +CPU=  9.732
  21:49:18 STEP 2924 H=2924:00 +CPU= 10.027
  21:49:34 STEP 2925 H=2925:00 +CPU= 15.672
  21:49:44 STEP 2926 H=2926:00 +CPU= 10.001
  21:49:54 STEP 2927 H=2927:00 +CPU=  9.987
  21:50:13 STEP 2928 H=2928:00 +CPU= 18.523
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_121.zip
Uploading the intermediate file: upload_file_121.zip
  21:50:23 STEP 2929 H=2929:00 +CPU= 10.433
  21:50:33 STEP 2930 H=2930:00 +CPU= 10.330
  21:50:49 STEP 2931 H=2931:00 +CPU= 15.836
  21:50:59 STEP 2932 H=2932:00 +CPU= 10.035
  21:51:09 STEP 2933 H=2933:00 +CPU= 10.176
  21:51:26 STEP 2934 H=2934:00 +CPU= 16.187
  21:51:35 STEP 2935 H=2935:00 +CPU=  9.739
  21:51:45 STEP 2936 H=2936:00 +CPU= 10.026
  21:52:02 STEP 2937 H=2937:00 +CPU= 16.472
  21:52:12 STEP 2938 H=2938:00 +CPU= 10.261
  21:52:22 STEP 2939 H=2939:00 +CPU=  9.782
  21:52:40 STEP 2940 H=2940:00 +CPU= 18.227
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002940
  21:52:50 STEP 2941 H=2941:00 +CPU=  9.995
  21:53:00 STEP 2942 H=2942:00 +CPU= 10.244
  21:53:16 STEP 2943 H=2943:00 +CPU= 16.096
  21:53:27 STEP 2944 H=2944:00 +CPU= 10.274
  21:53:37 STEP 2945 H=2945:00 +CPU= 10.045
  21:53:53 STEP 2946 H=2946:00 +CPU= 16.098
  21:54:03 STEP 2947 H=2947:00 +CPU=  9.721
  21:54:13 STEP 2948 H=2948:00 +CPU= 10.045
  21:54:29 STEP 2949 H=2949:00 +CPU= 16.416
  21:54:39 STEP 2950 H=2950:00 +CPU= 10.277
  21:54:49 STEP 2951 H=2951:00 +CPU=  9.994
  21:55:07 STEP 2952 H=2952:00 +CPU= 18.156
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/5/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/5/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/5/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12198588/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0944_2012050100_123_981_12198588_0_r218849604_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
21:58:31 (247): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:04:01 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 10,623,600 38,747 0.0036
09 May 2023 16:03:56 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 10,368,000 37,870 0.0037
09 May 2023 16:03:56 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 10,368,000 37,870 0.0037
09 May 2023 16:03:56 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 9,504,000 34,812 0.0037
09 May 2023 16:03:56 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 9,504,000 34,812 0.0037
09 May 2023 16:03:54 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 8,640,000 31,705 0.0037
09 May 2023 16:03:51 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 7,776,000 28,524 0.0037
09 May 2023 16:03:48 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 6,912,000 25,344 0.0037
09 May 2023 16:03:46 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 6,048,000 22,135 0.0037
09 May 2023 16:03:43 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 5,184,000 18,962 0.0037
09 May 2023 16:03:40 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 4,320,000 15,765 0.0036
09 May 2023 16:03:37 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 3,456,000 12,673 0.0037
09 May 2023 16:03:37 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 2,592,000 9,572 0.0037
09 May 2023 16:03:34 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 1,728,000 6,438 0.0037
09 May 2023 16:03:32 1537625 22284699 oifs_43r3_ps_0944_2012050100_123_981_12198588_0 864,000 3,292 0.0038


©2024 climateprediction.net