climateprediction.net home page
Task 22280751

Task 22280751

Name oifs_43r3_ps_0221_2009050100_123_978_12194865_0
Workunit 12194865
Created 23 Dec 2022, 20:29:39 UTC
Sent 23 Dec 2022, 21:09:59 UTC
Report deadline 22 Jan 2023, 21:09:59 UTC
Received 11 Jan 2023, 20:07:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537350
Run time 22 hours 32 min 44 sec
CPU time 22 hours 29 min 16 sec
Validate state Invalid
Credit 13,305.60
Device peak FLOPS 5.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.73 MB
Peak swap size 4,978.08 MB
Peak disk usage 1,348.05 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_99.zip
Uploading the intermediate file: upload_file_99.zip
  15:41:39 STEP 2401 H=2401:00 +CPU= 23.239
Uploading trickle at timestep: 8640000
  15:42:06 STEP 2402 H=2402:00 +CPU= 26.461
  15:42:45 STEP 2403 H=2403:00 +CPU= 39.077
  15:43:10 STEP 2404 H=2404:00 +CPU= 25.599
  15:43:35 STEP 2405 H=2405:00 +CPU= 24.502
  15:44:11 STEP 2406 H=2406:00 +CPU= 36.051
  15:44:33 STEP 2407 H=2407:00 +CPU= 22.023
  15:44:58 STEP 2408 H=2408:00 +CPU= 24.514
  15:45:37 STEP 2409 H=2409:00 +CPU= 39.114
  15:46:02 STEP 2410 H=2410:00 +CPU= 25.609
  15:46:29 STEP 2411 H=2411:00 +CPU= 26.215
  15:47:12 STEP 2412 H=2412:00 +CPU= 43.258
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002412
  15:47:37 STEP 2413 H=2413:00 +CPU= 24.950
  15:48:02 STEP 2414 H=2414:00 +CPU= 24.866
  15:48:39 STEP 2415 H=2415:00 +CPU= 36.786
  15:49:03 STEP 2416 H=2416:00 +CPU= 24.187
  15:49:28 STEP 2417 H=2417:00 +CPU= 25.261
  15:50:06 STEP 2418 H=2418:00 +CPU= 38.189
  15:50:31 STEP 2419 H=2419:00 +CPU= 24.470
  15:50:57 STEP 2420 H=2420:00 +CPU= 26.079
  15:51:37 STEP 2421 H=2421:00 +CPU= 39.874
  15:52:02 STEP 2422 H=2422:00 +CPU= 25.612
  15:52:29 STEP 2423 H=2423:00 +CPU= 26.173
  15:53:15 STEP 2424 H=2424:00 +CPU= 46.639
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_100.zip
Uploading the intermediate file: upload_file_100.zip
  15:53:41 STEP 2425 H=2425:00 +CPU= 25.544
  15:54:07 STEP 2426 H=2426:00 +CPU= 25.928
  15:54:45 STEP 2427 H=2427:00 +CPU= 37.834
  15:55:10 STEP 2428 H=2428:00 +CPU= 25.727
  15:55:34 STEP 2429 H=2429:00 +CPU= 24.205
  15:56:12 STEP 2430 H=2430:00 +CPU= 37.664
  15:56:38 STEP 2431 H=2431:00 +CPU= 25.303
  15:57:01 STEP 2432 H=2432:00 +CPU= 23.441
  15:57:40 STEP 2433 H=2433:00 +CPU= 38.391
  15:58:05 STEP 2434 H=2434:00 +CPU= 25.239
  15:58:27 STEP 2435 H=2435:00 +CPU= 21.969
  15:59:10 STEP 2436 H=2436:00 +CPU= 43.309
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002436
  15:59:35 STEP 2437 H=2437:00 +CPU= 24.443
  16:00:01 STEP 2438 H=2438:00 +CPU= 26.538
  16:00:41 STEP 2439 H=2439:00 +CPU= 39.856
  16:01:07 STEP 2440 H=2440:00 +CPU= 25.839
  16:01:32 STEP 2441 H=2441:00 +CPU= 25.435
  16:02:08 STEP 2442 H=2442:00 +CPU= 35.760
  16:02:34 STEP 2443 H=2443:00 +CPU= 25.547
  16:02:58 STEP 2444 H=2444:00 +CPU= 24.335
  16:03:36 STEP 2445 H=2445:00 +CPU= 38.169
  16:04:02 STEP 2446 H=2446:00 +CPU= 25.756
  16:04:27 STEP 2447 H=2447:00 +CPU= 25.347
  16:05:12 STEP 2448 H=2448:00 +CPU= 44.646
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_101.zip
Uploading the intermediate file: upload_file_101.zip
  16:05:37 STEP 2449 H=2449:00 +CPU= 25.550
  16:06:04 STEP 2450 H=2450:00 +CPU= 26.019
  16:06:43 STEP 2451 H=2451:00 +CPU= 39.077
  16:07:09 STEP 2452 H=2452:00 +CPU= 26.201
  16:07:34 STEP 2453 H=2453:00 +CPU= 25.096
  16:08:14 STEP 2454 H=2454:00 +CPU= 40.378
  16:08:40 STEP 2455 H=2455:00 +CPU= 25.255
  16:09:06 STEP 2456 H=2456:00 +CPU= 26.420
  16:09:45 STEP 2457 H=2457:00 +CPU= 39.060
  16:10:09 STEP 2458 H=2458:00 +CPU= 23.761
  16:10:34 STEP 2459 H=2459:00 +CPU= 25.244
  16:11:18 STEP 2460 H=2460:00 +CPU= 43.740
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002460
  16:11:43 STEP 2461 H=2461:00 +CPU= 25.399
  16:12:09 STEP 2462 H=2462:00 +CPU= 25.089
  16:12:44 STEP 2463 H=2463:00 +CPU= 35.612
  16:13:10 STEP 2464 H=2464:00 +CPU= 25.568
  16:13:35 STEP 2465 H=2465:00 +CPU= 25.374
  16:14:15 STEP 2466 H=2466:00 +CPU= 40.230
  16:14:40 STEP 2467 H=2467:00 +CPU= 25.074
  16:15:06 STEP 2468 H=2468:00 +CPU= 25.819
  16:15:46 STEP 2469 H=2469:00 +CPU= 39.773
  16:16:11 STEP 2470 H=2470:00 +CPU= 25.197
  16:16:37 STEP 2471 H=2471:00 +CPU= 25.605
  16:17:24 STEP 2472 H=2472:00 +CPU= 46.689
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_102.zip
Uploading the intermediate file: upload_file_102.zip
  16:17:49 STEP 2473 H=2473:00 +CPU= 25.252
  16:18:15 STEP 2474 H=2474:00 +CPU= 26.127
  16:18:55 STEP 2475 H=2475:00 +CPU= 39.875
  16:19:21 STEP 2476 H=2476:00 +CPU= 25.629
  16:19:44 STEP 2477 H=2477:00 +CPU= 23.527
  16:20:22 STEP 2478 H=2478:00 +CPU= 37.844
  16:20:44 STEP 2479 H=2479:00 +CPU= 22.400
  16:21:09 STEP 2480 H=2480:00 +CPU= 24.983
  16:21:45 STEP 2481 H=2481:00 +CPU= 35.995
  16:22:12 STEP 2482 H=2482:00 +CPU= 26.229
  16:22:36 STEP 2483 H=2483:00 +CPU= 23.831
  16:23:16 STEP 2484 H=2484:00 +CPU= 39.860
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002484
  16:23:40 STEP 2485 H=2485:00 +CPU= 24.103
  16:24:05 STEP 2486 H=2486:00 +CPU= 25.148
  16:24:43 STEP 2487 H=2487:00 +CPU= 38.223
  16:25:08 STEP 2488 H=2488:00 +CPU= 25.193
  16:25:33 STEP 2489 H=2489:00 +CPU= 24.803
  16:26:13 STEP 2490 H=2490:00 +CPU= 39.834
  16:26:38 STEP 2491 H=2491:00 +CPU= 25.223
  16:27:04 STEP 2492 H=2492:00 +CPU= 25.479
  16:27:42 STEP 2493 H=2493:00 +CPU= 38.195
  16:28:08 STEP 2494 H=2494:00 +CPU= 26.254
  16:28:30 STEP 2495 H=2495:00 +CPU= 21.889
  16:29:16 STEP 2496 H=2496:00 +CPU= 46.373
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_103.zip
Uploading the intermediate file: upload_file_103.zip
  16:29:40 STEP 2497 H=2497:00 +CPU= 23.893
  16:30:06 STEP 2498 H=2498:00 +CPU= 25.339
  16:30:44 STEP 2499 H=2499:00 +CPU= 38.378
  16:31:10 STEP 2500 H=2500:00 +CPU= 25.830
  16:31:35 STEP 2501 H=2501:00 +CPU= 25.181
  16:32:16 STEP 2502 H=2502:00 +CPU= 40.362
  16:32:41 STEP 2503 H=2503:00 +CPU= 25.359
  16:33:06 STEP 2504 H=2504:00 +CPU= 25.190
  16:33:44 STEP 2505 H=2505:00 +CPU= 37.390
  16:34:07 STEP 2506 H=2506:00 +CPU= 23.663
  16:34:26 STEP 2507 H=2507:00 +CPU= 19.087
  16:35:10 STEP 2508 H=2508:00 +CPU= 43.449
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002508
  16:35:35 STEP 2509 H=2509:00 +CPU= 25.102
  16:35:58 STEP 2510 H=2510:00 +CPU= 23.214
  16:36:37 STEP 2511 H=2511:00 +CPU= 38.424
  16:37:02 STEP 2512 H=2512:00 +CPU= 25.254
  16:37:24 STEP 2513 H=2513:00 +CPU= 22.365
  16:38:04 STEP 2514 H=2514:00 +CPU= 40.037
  16:38:30 STEP 2515 H=2515:00 +CPU= 25.868
  16:38:56 STEP 2516 H=2516:00 +CPU= 25.449
  16:39:35 STEP 2517 H=2517:00 +CPU= 39.340
  16:40:01 STEP 2518 H=2518:00 +CPU= 25.913
  16:40:27 STEP 2519 H=2519:00 +CPU= 26.018
  16:41:13 STEP 2520 H=2520:00 +CPU= 46.500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_104.zip
Uploading the intermediate file: upload_file_104.zip
  16:41:39 STEP 2521 H=2521:00 +CPU= 25.421
  16:42:04 STEP 2522 H=2522:00 +CPU= 24.906
  16:42:42 STEP 2523 H=2523:00 +CPU= 37.732
  16:43:07 STEP 2524 H=2524:00 +CPU= 25.042
  16:43:29 STEP 2525 H=2525:00 +CPU= 22.716
  16:44:03 STEP 2526 H=2526:00 +CPU= 33.304
  16:44:24 STEP 2527 H=2527:00 +CPU= 21.514
  16:44:49 STEP 2528 H=2528:00 +CPU= 25.205
  16:45:27 STEP 2529 H=2529:00 +CPU= 37.605
  16:45:52 STEP 2530 H=2530:00 +CPU= 24.633
  16:46:13 STEP 2531 H=2531:00 +CPU= 20.919
  16:46:53 STEP 2532 H=2532:00 +CPU= 40.113
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002532
  16:47:17 STEP 2533 H=2533:00 +CPU= 24.672
  16:47:42 STEP 2534 H=2534:00 +CPU= 24.839
  16:48:20 STEP 2535 H=2535:00 +CPU= 37.529
  16:48:46 STEP 2536 H=2536:00 +CPU= 25.793
  16:49:09 STEP 2537 H=2537:00 +CPU= 23.774
  16:49:48 STEP 2538 H=2538:00 +CPU= 38.396
  16:50:13 STEP 2539 H=2539:00 +CPU= 24.659
  16:50:38 STEP 2540 H=2540:00 +CPU= 25.038
  16:51:15 STEP 2541 H=2541:00 +CPU= 37.568
  16:51:41 STEP 2542 H=2542:00 +CPU= 25.456
  16:52:03 STEP 2543 H=2543:00 +CPU= 22.186
  16:52:45 STEP 2544 H=2544:00 +CPU= 41.532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_105.zip
Uploading the intermediate file: upload_file_105.zip
  16:53:09 STEP 2545 H=2545:00 +CPU= 24.416
  16:53:34 STEP 2546 H=2546:00 +CPU= 24.842
  16:54:13 STEP 2547 H=2547:00 +CPU= 38.696
  16:54:39 STEP 2548 H=2548:00 +CPU= 26.573
  16:55:05 STEP 2549 H=2549:00 +CPU= 25.482
  16:55:46 STEP 2550 H=2550:00 +CPU= 40.801
  16:56:11 STEP 2551 H=2551:00 +CPU= 25.701
  16:56:37 STEP 2552 H=2552:00 +CPU= 26.161
  16:57:14 STEP 2553 H=2553:00 +CPU= 36.675
  16:57:38 STEP 2554 H=2554:00 +CPU= 23.763
  16:58:03 STEP 2555 H=2555:00 +CPU= 25.046
  16:58:46 STEP 2556 H=2556:00 +CPU= 42.487
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002556
  16:59:12 STEP 2557 H=2557:00 +CPU= 25.985
  16:59:36 STEP 2558 H=2558:00 +CPU= 24.123
  17:00:13 STEP 2559 H=2559:00 +CPU= 37.114
  17:00:39 STEP 2560 H=2560:00 +CPU= 26.156
  17:01:02 STEP 2561 H=2561:00 +CPU= 23.230
  17:01:41 STEP 2562 H=2562:00 +CPU= 38.652
  17:02:07 STEP 2563 H=2563:00 +CPU= 26.417
  17:02:34 STEP 2564 H=2564:00 +CPU= 26.421
  17:03:13 STEP 2565 H=2565:00 +CPU= 39.206
  17:03:40 STEP 2566 H=2566:00 +CPU= 26.295
  17:04:06 STEP 2567 H=2567:00 +CPU= 26.415
  17:04:52 STEP 2568 H=2568:00 +CPU= 46.085
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_106.zip
Uploading the intermediate file: upload_file_106.zip
  17:05:17 STEP 2569 H=2569:00 +CPU= 25.378
  17:05:43 STEP 2570 H=2570:00 +CPU= 25.350
  17:06:21 STEP 2571 H=2571:00 +CPU= 37.957
  17:06:45 STEP 2572 H=2572:00 +CPU= 24.484
  17:07:10 STEP 2573 H=2573:00 +CPU= 24.267
  17:07:50 STEP 2574 H=2574:00 +CPU= 40.322
  17:08:16 STEP 2575 H=2575:00 +CPU= 26.154
  17:08:42 STEP 2576 H=2576:00 +CPU= 25.930
  17:09:21 STEP 2577 H=2577:00 +CPU= 38.746
  17:09:45 STEP 2578 H=2578:00 +CPU= 24.618
  17:10:10 STEP 2579 H=2579:00 +CPU= 24.938
  17:10:53 STEP 2580 H=2580:00 +CPU= 42.332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002580
  17:11:18 STEP 2581 H=2581:00 +CPU= 25.236
  17:11:43 STEP 2582 H=2582:00 +CPU= 25.198
  17:12:23 STEP 2583 H=2583:00 +CPU= 39.583
  17:12:49 STEP 2584 H=2584:00 +CPU= 26.294
  17:13:14 STEP 2585 H=2585:00 +CPU= 24.894
  17:13:54 STEP 2586 H=2586:00 +CPU= 40.117
  17:14:19 STEP 2587 H=2587:00 +CPU= 25.093
  17:14:45 STEP 2588 H=2588:00 +CPU= 25.777
  17:15:23 STEP 2589 H=2589:00 +CPU= 37.972
  17:15:49 STEP 2590 H=2590:00 +CPU= 25.740
  17:16:14 STEP 2591 H=2591:00 +CPU= 25.325
  17:17:00 STEP 2592 H=2592:00 +CPU= 45.829
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_107.zip
Uploading the intermediate file: upload_file_107.zip
  17:17:24 STEP 2593 H=2593:00 +CPU= 23.874
  17:17:48 STEP 2594 H=2594:00 +CPU= 23.697
  17:18:25 STEP 2595 H=2595:00 +CPU= 37.839
  17:18:50 STEP 2596 H=2596:00 +CPU= 24.045
  17:19:14 STEP 2597 H=2597:00 +CPU= 24.691
  17:19:54 STEP 2598 H=2598:00 +CPU= 40.007
  17:20:20 STEP 2599 H=2599:00 +CPU= 25.505
  17:20:45 STEP 2600 H=2600:00 +CPU= 24.856
  17:21:24 STEP 2601 H=2601:00 +CPU= 39.630
  17:21:50 STEP 2602 H=2602:00 +CPU= 25.925
  17:22:14 STEP 2603 H=2603:00 +CPU= 23.610
  17:22:55 STEP 2604 H=2604:00 +CPU= 41.413
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
  17:23:20 STEP 2605 H=2605:00 +CPU= 24.886
  17:23:45 STEP 2606 H=2606:00 +CPU= 25.125
  17:24:23 STEP 2607 H=2607:00 +CPU= 37.639
  17:24:47 STEP 2608 H=2608:00 +CPU= 23.667
  17:25:08 STEP 2609 H=2609:00 +CPU= 21.345
  17:25:47 STEP 2610 H=2610:00 +CPU= 39.198
  17:26:10 STEP 2611 H=2611:00 +CPU= 22.526
  17:26:34 STEP 2612 H=2612:00 +CPU= 24.163
  17:27:12 STEP 2613 H=2613:00 +CPU= 37.444
  17:27:36 STEP 2614 H=2614:00 +CPU= 23.992
  17:28:02 STEP 2615 H=2615:00 +CPU= 26.190
  17:28:48 STEP 2616 H=2616:00 +CPU= 46.165
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_108.zip
Uploading the intermediate file: upload_file_108.zip
  17:29:13 STEP 2617 H=2617:00 +CPU= 24.608
  17:29:39 STEP 2618 H=2618:00 +CPU= 25.963
  17:30:17 STEP 2619 H=2619:00 +CPU= 38.625
  17:30:43 STEP 2620 H=2620:00 +CPU= 25.867
  17:31:08 STEP 2621 H=2621:00 +CPU= 24.582
  17:31:48 STEP 2622 H=2622:00 +CPU= 39.876
  17:32:13 STEP 2623 H=2623:00 +CPU= 25.191
  17:32:37 STEP 2624 H=2624:00 +CPU= 24.491
  17:33:16 STEP 2625 H=2625:00 +CPU= 38.410
  17:33:42 STEP 2626 H=2626:00 +CPU= 25.792
  17:34:07 STEP 2627 H=2627:00 +CPU= 25.059
  17:34:50 STEP 2628 H=2628:00 +CPU= 43.039
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
  17:35:14 STEP 2629 H=2629:00 +CPU= 23.907
  17:35:39 STEP 2630 H=2630:00 +CPU= 25.265
  17:36:15 STEP 2631 H=2631:00 +CPU= 35.858
  17:36:41 STEP 2632 H=2632:00 +CPU= 26.034
  17:37:06 STEP 2633 H=2633:00 +CPU= 25.033
  17:37:43 STEP 2634 H=2634:00 +CPU= 37.408
  17:38:05 STEP 2635 H=2635:00 +CPU= 21.706
  17:38:28 STEP 2636 H=2636:00 +CPU= 23.158
  17:39:08 STEP 2637 H=2637:00 +CPU= 39.281
  17:39:34 STEP 2638 H=2638:00 +CPU= 26.518
  17:40:00 STEP 2639 H=2639:00 +CPU= 25.773
  17:40:45 STEP 2640 H=2640:00 +CPU= 44.776
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_109.zip
Uploading the intermediate file: upload_file_109.zip
Uploading trickle at timestep: 9504000
  17:41:10 STEP 2641 H=2641:00 +CPU= 25.607
  17:41:36 STEP 2642 H=2642:00 +CPU= 25.615
  17:42:15 STEP 2643 H=2643:00 +CPU= 38.956
  17:42:41 STEP 2644 H=2644:00 +CPU= 25.955
  17:43:05 STEP 2645 H=2645:00 +CPU= 23.895
  17:43:45 STEP 2646 H=2646:00 +CPU= 40.744
  17:44:11 STEP 2647 H=2647:00 +CPU= 25.950
  17:44:38 STEP 2648 H=2648:00 +CPU= 26.122
  17:45:17 STEP 2649 H=2649:00 +CPU= 39.865
  17:45:44 STEP 2650 H=2650:00 +CPU= 26.603
  17:46:08 STEP 2651 H=2651:00 +CPU= 24.339
  17:46:49 STEP 2652 H=2652:00 +CPU= 40.418
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652
  17:47:10 STEP 2653 H=2653:00 +CPU= 20.696
  17:47:35 STEP 2654 H=2654:00 +CPU= 25.526
  17:48:15 STEP 2655 H=2655:00 +CPU= 40.007
  17:48:41 STEP 2656 H=2656:00 +CPU= 26.311
  17:49:07 STEP 2657 H=2657:00 +CPU= 25.526
  17:49:47 STEP 2658 H=2658:00 +CPU= 39.980
  17:50:09 STEP 2659 H=2659:00 +CPU= 22.343
  17:50:33 STEP 2660 H=2660:00 +CPU= 23.919
  17:51:12 STEP 2661 H=2661:00 +CPU= 38.593
  17:51:38 STEP 2662 H=2662:00 +CPU= 26.537
  17:52:04 STEP 2663 H=2663:00 +CPU= 25.290
  17:52:48 STEP 2664 H=2664:00 +CPU= 44.600
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_110.zip
Uploading the intermediate file: upload_file_110.zip
  17:53:13 STEP 2665 H=2665:00 +CPU= 24.826
  17:53:38 STEP 2666 H=2666:00 +CPU= 24.904
  17:54:15 STEP 2667 H=2667:00 +CPU= 36.491
  17:54:39 STEP 2668 H=2668:00 +CPU= 24.348
  17:55:03 STEP 2669 H=2669:00 +CPU= 24.298
  17:55:43 STEP 2670 H=2670:00 +CPU= 40.013
  17:56:09 STEP 2671 H=2671:00 +CPU= 25.483
  17:56:35 STEP 2672 H=2672:00 +CPU= 26.343
  17:57:14 STEP 2673 H=2673:00 +CPU= 38.588
  17:57:40 STEP 2674 H=2674:00 +CPU= 26.252
  17:58:05 STEP 2675 H=2675:00 +CPU= 24.820
  17:58:49 STEP 2676 H=2676:00 +CPU= 44.084
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676
  17:59:15 STEP 2677 H=2677:00 +CPU= 25.662
  17:59:40 STEP 2678 H=2678:00 +CPU= 25.600
  18:00:20 STEP 2679 H=2679:00 +CPU= 39.538
  18:00:46 STEP 2680 H=2680:00 +CPU= 25.901
  18:01:11 STEP 2681 H=2681:00 +CPU= 25.091
  18:01:51 STEP 2682 H=2682:00 +CPU= 39.977
  18:02:16 STEP 2683 H=2683:00 +CPU= 25.008
  18:02:42 STEP 2684 H=2684:00 +CPU= 25.796
  18:03:20 STEP 2685 H=2685:00 +CPU= 38.870
  18:03:46 STEP 2686 H=2686:00 +CPU= 25.049
  18:04:07 STEP 2687 H=2687:00 +CPU= 21.485
  18:04:52 STEP 2688 H=2688:00 +CPU= 45.080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_111.zip
Uploading the intermediate file: upload_file_111.zip
  18:05:14 STEP 2689 H=2689:00 +CPU= 21.472
  18:05:37 STEP 2690 H=2690:00 +CPU= 23.342
  18:06:15 STEP 2691 H=2691:00 +CPU= 37.524
  18:06:40 STEP 2692 H=2692:00 +CPU= 25.732
  18:07:04 STEP 2693 H=2693:00 +CPU= 23.817
  18:07:42 STEP 2694 H=2694:00 +CPU= 37.662
  18:08:07 STEP 2695 H=2695:00 +CPU= 25.030
  18:08:30 STEP 2696 H=2696:00 +CPU= 23.404
  18:09:09 STEP 2697 H=2697:00 +CPU= 39.043
  18:09:35 STEP 2698 H=2698:00 +CPU= 25.352
  18:09:58 STEP 2699 H=2699:00 +CPU= 23.646
  18:10:43 STEP 2700 H=2700:00 +CPU= 44.182
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700
  18:11:07 STEP 2701 H=2701:00 +CPU= 24.354
  18:11:32 STEP 2702 H=2702:00 +CPU= 25.027
  18:12:10 STEP 2703 H=2703:00 +CPU= 38.288
  18:12:35 STEP 2704 H=2704:00 +CPU= 24.487
  18:12:58 STEP 2705 H=2705:00 +CPU= 23.613
  18:13:38 STEP 2706 H=2706:00 +CPU= 39.266
  18:14:01 STEP 2707 H=2707:00 +CPU= 23.137
  18:14:26 STEP 2708 H=2708:00 +CPU= 25.269
  18:15:06 STEP 2709 H=2709:00 +CPU= 39.800
  18:15:31 STEP 2710 H=2710:00 +CPU= 25.294
  18:15:58 STEP 2711 H=2711:00 +CPU= 26.164
  18:16:44 STEP 2712 H=2712:00 +CPU= 46.697
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_112.zip
Uploading the intermediate file: upload_file_112.zip
  18:17:09 STEP 2713 H=2713:00 +CPU= 25.188
  18:17:35 STEP 2714 H=2714:00 +CPU= 25.942
  18:18:14 STEP 2715 H=2715:00 +CPU= 39.010
  18:18:41 STEP 2716 H=2716:00 +CPU= 26.196
  18:19:05 STEP 2717 H=2717:00 +CPU= 24.086
  18:19:43 STEP 2718 H=2718:00 +CPU= 38.285
  18:20:08 STEP 2719 H=2719:00 +CPU= 25.355
  18:20:34 STEP 2720 H=2720:00 +CPU= 25.950
  18:21:14 STEP 2721 H=2721:00 +CPU= 39.362
  18:21:40 STEP 2722 H=2722:00 +CPU= 25.990
  18:22:05 STEP 2723 H=2723:00 +CPU= 25.247
  18:22:47 STEP 2724 H=2724:00 +CPU= 42.101
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002724
  18:23:12 STEP 2725 H=2725:00 +CPU= 24.724
  18:23:38 STEP 2726 H=2726:00 +CPU= 25.721
  18:24:16 STEP 2727 H=2727:00 +CPU= 38.745
  18:24:39 STEP 2728 H=2728:00 +CPU= 22.777
  18:25:01 STEP 2729 H=2729:00 +CPU= 22.101
  18:25:38 STEP 2730 H=2730:00 +CPU= 36.835
  18:26:03 STEP 2731 H=2731:00 +CPU= 24.604
  18:26:27 STEP 2732 H=2732:00 +CPU= 24.674
  18:27:05 STEP 2733 H=2733:00 +CPU= 37.604
  18:27:31 STEP 2734 H=2734:00 +CPU= 25.802
  18:27:57 STEP 2735 H=2735:00 +CPU= 25.996
  18:28:44 STEP 2736 H=2736:00 +CPU= 46.884
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_113.zip
Uploading the intermediate file: upload_file_113.zip
  18:29:09 STEP 2737 H=2737:00 +CPU= 25.407
  18:29:35 STEP 2738 H=2738:00 +CPU= 25.790
  18:30:14 STEP 2739 H=2739:00 +CPU= 39.194
  18:30:41 STEP 2740 H=2740:00 +CPU= 26.370
  18:31:05 STEP 2741 H=2741:00 +CPU= 24.527
  18:31:44 STEP 2742 H=2742:00 +CPU= 38.670
  18:32:09 STEP 2743 H=2743:00 +CPU= 25.324
  18:32:35 STEP 2744 H=2744:00 +CPU= 25.539
  18:33:12 STEP 2745 H=2745:00 +CPU= 37.027
  18:33:37 STEP 2746 H=2746:00 +CPU= 25.198
  18:33:58 STEP 2747 H=2747:00 +CPU= 20.848
  18:34:39 STEP 2748 H=2748:00 +CPU= 40.704
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002748
  18:35:03 STEP 2749 H=2749:00 +CPU= 24.096
  18:35:24 STEP 2750 H=2750:00 +CPU= 21.594
  18:35:55 STEP 2751 H=2751:00 +CPU= 31.118
  18:36:18 STEP 2752 H=2752:00 +CPU= 22.664
  18:36:40 STEP 2753 H=2753:00 +CPU= 21.655
  18:37:17 STEP 2754 H=2754:00 +CPU= 36.819
  18:37:42 STEP 2755 H=2755:00 +CPU= 24.951
  18:38:07 STEP 2756 H=2756:00 +CPU= 25.694
  18:38:47 STEP 2757 H=2757:00 +CPU= 39.618
  18:39:12 STEP 2758 H=2758:00 +CPU= 25.011
  18:39:38 STEP 2759 H=2759:00 +CPU= 25.688
  18:40:22 STEP 2760 H=2760:00 +CPU= 44.768
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_114.zip
Uploading the intermediate file: upload_file_114.zip
  18:40:47 STEP 2761 H=2761:00 +CPU= 24.141
  18:41:08 STEP 2762 H=2762:00 +CPU= 21.215
  18:41:45 STEP 2763 H=2763:00 +CPU= 36.801
  18:42:10 STEP 2764 H=2764:00 +CPU= 25.363
  18:42:32 STEP 2765 H=2765:00 +CPU= 21.887
  18:43:11 STEP 2766 H=2766:00 +CPU= 38.903
  18:43:35 STEP 2767 H=2767:00 +CPU= 24.223
  18:44:00 STEP 2768 H=2768:00 +CPU= 25.141
  18:44:38 STEP 2769 H=2769:00 +CPU= 38.209
  18:45:02 STEP 2770 H=2770:00 +CPU= 23.613
  18:45:20 STEP 2771 H=2771:00 +CPU= 17.867
  18:45:53 STEP 2772 H=2772:00 +CPU= 32.712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002772
  18:46:18 STEP 2773 H=2773:00 +CPU= 25.042
  18:46:42 STEP 2774 H=2774:00 +CPU= 24.761
  18:47:19 STEP 2775 H=2775:00 +CPU= 36.660
  18:47:45 STEP 2776 H=2776:00 +CPU= 25.895
  18:48:10 STEP 2777 H=2777:00 +CPU= 24.824
  18:48:47 STEP 2778 H=2778:00 +CPU= 37.118
  18:49:12 STEP 2779 H=2779:00 +CPU= 24.947
  18:49:37 STEP 2780 H=2780:00 +CPU= 24.803
  18:50:15 STEP 2781 H=2781:00 +CPU= 38.569
  18:50:39 STEP 2782 H=2782:00 +CPU= 23.482
  18:51:04 STEP 2783 H=2783:00 +CPU= 25.047
  18:51:50 STEP 2784 H=2784:00 +CPU= 45.831
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_115.zip
Uploading the intermediate file: upload_file_115.zip
  18:52:15 STEP 2785 H=2785:00 +CPU= 25.222
  18:52:41 STEP 2786 H=2786:00 +CPU= 25.800
  18:53:20 STEP 2787 H=2787:00 +CPU= 39.107
  18:53:46 STEP 2788 H=2788:00 +CPU= 26.440
  18:54:11 STEP 2789 H=2789:00 +CPU= 24.816
  18:54:50 STEP 2790 H=2790:00 +CPU= 39.253
  18:55:14 STEP 2791 H=2791:00 +CPU= 23.295
  18:55:39 STEP 2792 H=2792:00 +CPU= 24.849
  18:56:14 STEP 2793 H=2793:00 +CPU= 34.812
  18:56:40 STEP 2794 H=2794:00 +CPU= 26.612
  18:57:06 STEP 2795 H=2795:00 +CPU= 25.577
  18:57:47 STEP 2796 H=2796:00 +CPU= 40.922
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002796
  18:58:11 STEP 2797 H=2797:00 +CPU= 24.414
  18:58:37 STEP 2798 H=2798:00 +CPU= 25.561
  18:59:15 STEP 2799 H=2799:00 +CPU= 37.703
  18:59:41 STEP 2800 H=2800:00 +CPU= 26.582
  19:00:05 STEP 2801 H=2801:00 +CPU= 24.335
  19:00:45 STEP 2802 H=2802:00 +CPU= 39.287
  19:01:11 STEP 2803 H=2803:00 +CPU= 25.907
  19:01:37 STEP 2804 H=2804:00 +CPU= 26.132
  19:02:13 STEP 2805 H=2805:00 +CPU= 36.539
  19:02:39 STEP 2806 H=2806:00 +CPU= 25.828
  19:03:04 STEP 2807 H=2807:00 +CPU= 25.050
  19:03:47 STEP 2808 H=2808:00 +CPU= 42.697
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:04:12 STEP 2809 H=2809:00 +CPU= 24.915
  19:04:38 STEP 2810 H=2810:00 +CPU= 26.027
  19:05:17 STEP 2811 H=2811:00 +CPU= 38.509
  19:05:43 STEP 2812 H=2812:00 +CPU= 26.284
  19:06:08 STEP 2813 H=2813:00 +CPU= 25.133
  19:06:48 STEP 2814 H=2814:00 +CPU= 39.415
  19:07:14 STEP 2815 H=2815:00 +CPU= 26.044
  19:07:40 STEP 2816 H=2816:00 +CPU= 25.904
  19:08:18 STEP 2817 H=2817:00 +CPU= 38.765
  19:08:44 STEP 2818 H=2818:00 +CPU= 25.918
  19:09:09 STEP 2819 H=2819:00 +CPU= 24.504
  19:09:51 STEP 2820 H=2820:00 +CPU= 42.654
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820
  19:10:16 STEP 2821 H=2821:00 +CPU= 24.698
  19:10:43 STEP 2822 H=2822:00 +CPU= 26.640
  19:11:21 STEP 2823 H=2823:00 +CPU= 37.950
  19:11:46 STEP 2824 H=2824:00 +CPU= 25.087
  19:12:11 STEP 2825 H=2825:00 +CPU= 25.057
  19:12:50 STEP 2826 H=2826:00 +CPU= 39.376
  19:13:15 STEP 2827 H=2827:00 +CPU= 24.305
  19:13:40 STEP 2828 H=2828:00 +CPU= 25.745
  19:14:16 STEP 2829 H=2829:00 +CPU= 35.732
  19:14:42 STEP 2830 H=2830:00 +CPU= 25.866
  19:15:07 STEP 2831 H=2831:00 +CPU= 25.259
  19:15:53 STEP 2832 H=2832:00 +CPU= 45.553
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_117.zip
Uploading the intermediate file: upload_file_117.zip
  19:16:18 STEP 2833 H=2833:00 +CPU= 25.084
  19:16:44 STEP 2834 H=2834:00 +CPU= 26.087
  19:17:22 STEP 2835 H=2835:00 +CPU= 37.244
  19:17:46 STEP 2836 H=2836:00 +CPU= 24.890
  19:18:09 STEP 2837 H=2837:00 +CPU= 22.824
  19:18:48 STEP 2838 H=2838:00 +CPU= 38.481
  19:19:12 STEP 2839 H=2839:00 +CPU= 23.763
  19:19:37 STEP 2840 H=2840:00 +CPU= 24.935
  19:20:14 STEP 2841 H=2841:00 +CPU= 37.159
  19:20:38 STEP 2842 H=2842:00 +CPU= 24.026
  19:21:01 STEP 2843 H=2843:00 +CPU= 23.633
  19:21:43 STEP 2844 H=2844:00 +CPU= 41.422
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002844
  19:22:08 STEP 2845 H=2845:00 +CPU= 24.866
  19:22:33 STEP 2846 H=2846:00 +CPU= 24.765
  19:23:07 STEP 2847 H=2847:00 +CPU= 33.877
  19:23:31 STEP 2848 H=2848:00 +CPU= 24.670
  19:23:56 STEP 2849 H=2849:00 +CPU= 24.497
  19:24:34 STEP 2850 H=2850:00 +CPU= 38.614
  19:25:00 STEP 2851 H=2851:00 +CPU= 25.379
  19:25:28 STEP 2852 H=2852:00 +CPU= 28.002
  19:26:05 STEP 2853 H=2853:00 +CPU= 37.023
  19:26:31 STEP 2854 H=2854:00 +CPU= 25.687
  19:26:57 STEP 2855 H=2855:00 +CPU= 26.572
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4031MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4031 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-RADLSW       (1210) 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               RADIATION_SCHEME 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                radiation_interface:radiation 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192716:1671910036:80221.693] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 radiation_mcica_sw:solver_mcica_sw 
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192717:1671910037:80222.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:3862:3862] [20221224:192717:1671910037:80222.693] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001968EB9  radiation_mcica_s         189  radiation_mcica_sw.F90
oifs_43r3_model.e  0000000001862DC8  radiation_interfa         419  radiation_interface.F90
oifs_43r3_model.e  00000000012D30E5  radiation_scheme_         484  radiation_scheme.F90
oifs_43r3_model.e  000000000129A415  radintg_                 1455  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0221
(argv4) batchid: 978
(argv5) wuid: 12194865
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03450f04ffb021d3add631b0c3d22f05 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0221_2009050100_123_978_12194865.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0221_2009050100_123_978_12194865.zip
ic_ancil_file: ic_ancil_12194865
ifsdata_file: ifsdata_12194865
climate_data_file: clim_data_12194865
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4ad5c0da212213db4e22585cf8364597 to: /var/lib/boinc-client/slots/6/ic_ancil_12194865.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12194865.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fd065d8c7dbc2d5c72de80fa9da85fdf to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12194865.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12194865.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12194865.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12194865.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12194865.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12194865.xml
-- Model is restarting --
Adjusting last_iter, 2856, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12194865.xml
last_cpu_time: 80166
upload_file_number: 118
last_iter: 2855
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215
The child process has been launched with process id: 1149
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1149:1149] [20230111:185249:1673463169:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:54:41 STEP 2856 H=2856:00 +CPU= 47.286
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194865/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0221_2009050100_123_978_12194865_0_r2083661215_118.zip
Uploading the intermediate file: upload_file_118.zip
  18:55:06 STEP 2857 H=2857:00 +CPU= 25.007
  18:55:34 STEP 2858 H=2858:00 +CPU= 27.949
  18:56:16 STEP 2859 H=2859:00 +CPU= 42.451
  18:56:44 STEP 2860 H=2860:00 +CPU= 27.545
  18:57:11 STEP 2861 H=2861:00 +CPU= 27.159
  18:57:54 STEP 2862 H=2862:00 +CPU= 43.118
  18:58:22 STEP 2863 H=2863:00 +CPU= 27.558
  18:58:50 STEP 2864 H=2864:00 +CPU= 28.072
  18:59:33 STEP 2865 H=2865:00 +CPU= 42.619
  19:00:01 STEP 2866 H=2866:00 +CPU= 27.906
  19:00:28 STEP 2867 H=2867:00 +CPU= 27.142
  19:01:15 STEP 2868 H=2868:00 +CPU= 47.513
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002868
  19:01:43 STEP 2869 H=2869:00 +CPU= 27.731
  19:02:11 STEP 2870 H=2870:00 +CPU= 27.526
  19:02:53 STEP 2871 H=2871:00 +CPU= 42.566
  19:03:20 STEP 2872 H=2872:00 +CPU= 26.309
  19:03:46 STEP 2873 H=2873:00 +CPU= 25.999
  19:04:28 STEP 2874 H=2874:00 +CPU= 42.707
  19:04:56 STEP 2875 H=2875:00 +CPU= 27.654
  19:05:24 STEP 2876 H=2876:00 +CPU= 28.042
  19:06:05 STEP 2877 H=2877:00 +CPU= 41.468
  19:06:34 STEP 2878 H=2878:00 +CPU= 28.559
forrtl: No space left on device
forrtl: severe (38): error during write, unit 140, file /var/lib/boinc-client/slots/6/./LAW20090501000000_000119230000.1_1
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CD9519  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001D38C59  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001A02B77  writestress_               85  writestress.F
oifs_43r3_model.e  00000000016A224E  savstress_                163  savstress.F
oifs_43r3_model.e  0000000001693CF7  wamodel_                 1451  wamodel.F
oifs_43r3_model.e  0000000000872295  wavemdl_                  584  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:52:56 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 9,504,000 73,788 0.0078
09 May 2023 15:52:43 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 8,640,000 66,630 0.0077
09 May 2023 15:52:31 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 7,776,000 59,472 0.0076
09 May 2023 15:52:19 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 6,912,000 52,243 0.0076
09 May 2023 15:52:08 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 6,048,000 45,115 0.0075
09 May 2023 15:51:56 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 5,184,000 37,906 0.0073
09 May 2023 15:51:46 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 4,320,000 30,679 0.0071
09 May 2023 15:51:35 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 3,456,000 23,510 0.0068
09 May 2023 15:51:24 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 2,592,000 16,423 0.0063
09 May 2023 15:51:14 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 1,728,000 9,901 0.0057
09 May 2023 15:51:08 1537350 22280751 oifs_43r3_ps_0221_2009050100_123_978_12194865_0 864,000 4,593 0.0053


©2024 climateprediction.net