climateprediction.net home page
Task 22296420

Task 22296420

Name oifs_43r3_ps_0026_1987050100_123_956_12172670_1
Workunit 12172670
Created 8 Jan 2023, 14:52:28 UTC
Sent 8 Jan 2023, 15:15:12 UTC
Report deadline 7 Feb 2023, 15:15:12 UTC
Received 14 Jan 2023, 10:32:06 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1513601
Run time 15 hours 34 min 18 sec
CPU time 15 hours 24 min 55 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.14 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.25 MB
Peak swap size 4,978.08 MB
Peak disk usage 4,525.91 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
2484
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002484
  06:57:35 STEP 2485 H=2485:00 +CPU= 15.583
  06:57:51 STEP 2486 H=2486:00 +CPU= 15.832
  06:58:15 STEP 2487 H=2487:00 +CPU= 23.805
  06:58:31 STEP 2488 H=2488:00 +CPU= 15.777
  06:58:46 STEP 2489 H=2489:00 +CPU= 15.321
  06:59:11 STEP 2490 H=2490:00 +CPU= 24.491
  06:59:26 STEP 2491 H=2491:00 +CPU= 15.204
  06:59:42 STEP 2492 H=2492:00 +CPU= 15.732
  07:00:06 STEP 2493 H=2493:00 +CPU= 23.876
  07:00:22 STEP 2494 H=2494:00 +CPU= 15.642
  07:00:38 STEP 2495 H=2495:00 +CPU= 15.890
  07:01:06 STEP 2496 H=2496:00 +CPU= 28.019
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_103.zip
Uploading the intermediate file: upload_file_103.zip
  07:01:21 STEP 2497 H=2497:00 +CPU= 15.291
  07:01:37 STEP 2498 H=2498:00 +CPU= 15.732
  07:02:01 STEP 2499 H=2499:00 +CPU= 23.989
  07:02:17 STEP 2500 H=2500:00 +CPU= 15.860
  07:02:33 STEP 2501 H=2501:00 +CPU= 15.415
  07:02:57 STEP 2502 H=2502:00 +CPU= 24.342
  07:03:13 STEP 2503 H=2503:00 +CPU= 15.602
  07:03:29 STEP 2504 H=2504:00 +CPU= 15.840
  07:03:53 STEP 2505 H=2505:00 +CPU= 24.013
  07:04:09 STEP 2506 H=2506:00 +CPU= 15.909
  07:04:25 STEP 2507 H=2507:00 +CPU= 15.439
  07:04:51 STEP 2508 H=2508:00 +CPU= 26.257
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002508
  07:05:07 STEP 2509 H=2509:00 +CPU= 15.589
  07:05:23 STEP 2510 H=2510:00 +CPU= 15.796
  07:05:47 STEP 2511 H=2511:00 +CPU= 23.868
  07:06:03 STEP 2512 H=2512:00 +CPU= 15.814
  07:06:18 STEP 2513 H=2513:00 +CPU= 15.499
  07:06:43 STEP 2514 H=2514:00 +CPU= 24.212
  07:06:58 STEP 2515 H=2515:00 +CPU= 15.340
  07:07:14 STEP 2516 H=2516:00 +CPU= 15.736
  07:07:38 STEP 2517 H=2517:00 +CPU= 23.965
  07:07:54 STEP 2518 H=2518:00 +CPU= 15.615
  07:08:09 STEP 2519 H=2519:00 +CPU= 15.757
  07:08:38 STEP 2520 H=2520:00 +CPU= 28.524
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_104.zip
Uploading the intermediate file: upload_file_104.zip
  07:08:53 STEP 2521 H=2521:00 +CPU= 15.190
  07:09:09 STEP 2522 H=2522:00 +CPU= 15.830
  07:09:33 STEP 2523 H=2523:00 +CPU= 23.910
  07:09:49 STEP 2524 H=2524:00 +CPU= 15.850
  07:10:05 STEP 2525 H=2525:00 +CPU= 15.454
  07:10:29 STEP 2526 H=2526:00 +CPU= 24.095
  07:10:45 STEP 2527 H=2527:00 +CPU= 15.530
  07:11:01 STEP 2528 H=2528:00 +CPU= 15.759
  07:11:25 STEP 2529 H=2529:00 +CPU= 24.029
  07:11:41 STEP 2530 H=2530:00 +CPU= 15.799
  07:11:56 STEP 2531 H=2531:00 +CPU= 15.430
  07:12:23 STEP 2532 H=2532:00 +CPU= 26.311
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002532
  07:12:38 STEP 2533 H=2533:00 +CPU= 15.429
  07:12:54 STEP 2534 H=2534:00 +CPU= 15.835
  07:13:18 STEP 2535 H=2535:00 +CPU= 23.770
  07:13:34 STEP 2536 H=2536:00 +CPU= 15.808
  07:13:49 STEP 2537 H=2537:00 +CPU= 15.397
  07:14:14 STEP 2538 H=2538:00 +CPU= 24.342
  07:14:29 STEP 2539 H=2539:00 +CPU= 15.683
  07:14:45 STEP 2540 H=2540:00 +CPU= 15.893
  07:15:09 STEP 2541 H=2541:00 +CPU= 23.730
  07:15:25 STEP 2542 H=2542:00 +CPU= 15.713
  07:15:41 STEP 2543 H=2543:00 +CPU= 15.920
  07:16:10 STEP 2544 H=2544:00 +CPU= 28.144
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_105.zip
Uploading the intermediate file: upload_file_105.zip
  07:16:25 STEP 2545 H=2545:00 +CPU= 15.348
Uploading trickle at timestep: 9158400
  07:16:41 STEP 2546 H=2546:00 +CPU= 15.531
  07:17:04 STEP 2547 H=2547:00 +CPU= 23.252
  07:17:20 STEP 2548 H=2548:00 +CPU= 15.796
  07:17:35 STEP 2549 H=2549:00 +CPU= 15.253
  07:17:59 STEP 2550 H=2550:00 +CPU= 24.128
  07:18:15 STEP 2551 H=2551:00 +CPU= 15.452
  07:18:31 STEP 2552 H=2552:00 +CPU= 15.700
  07:18:54 STEP 2553 H=2553:00 +CPU= 23.556
  07:19:10 STEP 2554 H=2554:00 +CPU= 15.789
  07:19:26 STEP 2555 H=2555:00 +CPU= 15.603
  07:19:52 STEP 2556 H=2556:00 +CPU= 26.318
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002556
  07:20:08 STEP 2557 H=2557:00 +CPU= 15.174
  07:20:24 STEP 2558 H=2558:00 +CPU= 15.738
  07:20:47 STEP 2559 H=2559:00 +CPU= 23.407
  07:21:03 STEP 2560 H=2560:00 +CPU= 15.750
  07:21:18 STEP 2561 H=2561:00 +CPU= 15.278
  07:21:43 STEP 2562 H=2562:00 +CPU= 24.420
  07:21:58 STEP 2563 H=2563:00 +CPU= 15.431
  07:22:14 STEP 2564 H=2564:00 +CPU= 15.755
  07:22:38 STEP 2565 H=2565:00 +CPU= 23.558
  07:22:54 STEP 2566 H=2566:00 +CPU= 15.887
  07:23:10 STEP 2567 H=2567:00 +CPU= 15.868
  07:23:38 STEP 2568 H=2568:00 +CPU= 27.759
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072345:1673591025:45571.663] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   VDFHGHTN 
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072346:1673591026:45572.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:crunchy64:1:1:2202:2202] [20230113:072346:1673591026:45572.663] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FB7BBD  vdfhghtn_                 659  vdfhghtn.F90
oifs_43r3_model.e  0000000000FAB4D3  vdfmain_                  971  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0026
(argv4) batchid: 956
(argv5) wuid: 12172670
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1296b9c86a8b11d2d28fee4fe1a0e88b to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0026_1987050100_123_956_12172670.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0026_1987050100_123_956_12172670.zip
ic_ancil_file: ic_ancil_12172670
ifsdata_file: ifsdata_12172670
climate_data_file: clim_data_12172670
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_961885aa79212bc58311329f9a13a444 to: /var/lib/boinc-client/slots/2/ic_ancil_12172670.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12172670.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b22adb0df2395d5b4b6aa6c5dcf8b0a2 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12172670.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12172670.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12172670.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12172670.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12172670.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12172670.xml
-- Model is restarting --
Adjusting last_iter, 2568, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12172670.xml
last_cpu_time: 48282
upload_file_number: 106
last_iter: 2567
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992
The child process has been launched with process id: 2190
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:crunchy64:1:1:2190:2190] [20230114:081141:1673680301:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:12:17 STEP 2568 H=2568:00 +CPU= 26.632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_106.zip
Uploading the intermediate file: upload_file_106.zip
  08:12:33 STEP 2569 H=2569:00 +CPU= 14.823
  08:12:48 STEP 2570 H=2570:00 +CPU= 15.093
  08:13:11 STEP 2571 H=2571:00 +CPU= 23.371
  08:13:27 STEP 2572 H=2572:00 +CPU= 15.826
  08:13:43 STEP 2573 H=2573:00 +CPU= 15.295
  08:14:07 STEP 2574 H=2574:00 +CPU= 24.208
  08:14:22 STEP 2575 H=2575:00 +CPU= 15.229
  08:14:38 STEP 2576 H=2576:00 +CPU= 15.694
  08:15:02 STEP 2577 H=2577:00 +CPU= 23.505
  08:15:17 STEP 2578 H=2578:00 +CPU= 15.607
  08:15:33 STEP 2579 H=2579:00 +CPU= 15.348
  08:15:59 STEP 2580 H=2580:00 +CPU= 26.156
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002580
  08:16:14 STEP 2581 H=2581:00 +CPU= 15.085
  08:16:30 STEP 2582 H=2582:00 +CPU= 15.586
  08:16:53 STEP 2583 H=2583:00 +CPU= 23.235
  08:17:09 STEP 2584 H=2584:00 +CPU= 15.708
  08:17:25 STEP 2585 H=2585:00 +CPU= 15.347
  08:17:48 STEP 2586 H=2586:00 +CPU= 23.707
  08:18:04 STEP 2587 H=2587:00 +CPU= 15.304
  08:18:19 STEP 2588 H=2588:00 +CPU= 15.520
  08:18:43 STEP 2589 H=2589:00 +CPU= 23.553
  08:18:59 STEP 2590 H=2590:00 +CPU= 15.733
  08:19:15 STEP 2591 H=2591:00 +CPU= 15.725
  08:19:43 STEP 2592 H=2592:00 +CPU= 27.874
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_107.zip
Uploading the intermediate file: upload_file_107.zip
  08:19:58 STEP 2593 H=2593:00 +CPU= 15.418
  08:20:14 STEP 2594 H=2594:00 +CPU= 15.480
  08:20:37 STEP 2595 H=2595:00 +CPU= 23.476
  08:20:53 STEP 2596 H=2596:00 +CPU= 15.937
  08:21:09 STEP 2597 H=2597:00 +CPU= 15.463
  08:21:33 STEP 2598 H=2598:00 +CPU= 24.260
  08:21:49 STEP 2599 H=2599:00 +CPU= 15.555
  08:22:05 STEP 2600 H=2600:00 +CPU= 15.702
  08:22:28 STEP 2601 H=2601:00 +CPU= 23.525
  08:22:44 STEP 2602 H=2602:00 +CPU= 15.926
  08:23:00 STEP 2603 H=2603:00 +CPU= 15.531
  08:23:27 STEP 2604 H=2604:00 +CPU= 26.635
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002604
  08:23:42 STEP 2605 H=2605:00 +CPU= 15.394
  08:23:58 STEP 2606 H=2606:00 +CPU= 15.938
  08:24:22 STEP 2607 H=2607:00 +CPU= 23.586
  08:24:38 STEP 2608 H=2608:00 +CPU= 15.970
  08:24:53 STEP 2609 H=2609:00 +CPU= 15.395
  08:25:18 STEP 2610 H=2610:00 +CPU= 24.206
  08:25:33 STEP 2611 H=2611:00 +CPU= 15.483
  08:25:49 STEP 2612 H=2612:00 +CPU= 15.893
  08:26:13 STEP 2613 H=2613:00 +CPU= 23.929
  08:26:29 STEP 2614 H=2614:00 +CPU= 15.619
  08:26:45 STEP 2615 H=2615:00 +CPU= 15.679
  08:27:13 STEP 2616 H=2616:00 +CPU= 28.207
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_108.zip
Uploading the intermediate file: upload_file_108.zip
  08:27:29 STEP 2617 H=2617:00 +CPU= 15.489
  08:27:45 STEP 2618 H=2618:00 +CPU= 15.852
  08:28:09 STEP 2619 H=2619:00 +CPU= 23.698
  08:28:24 STEP 2620 H=2620:00 +CPU= 15.814
  08:28:40 STEP 2621 H=2621:00 +CPU= 15.379
  08:29:05 STEP 2622 H=2622:00 +CPU= 24.506
  08:29:20 STEP 2623 H=2623:00 +CPU= 15.628
  08:29:36 STEP 2624 H=2624:00 +CPU= 15.863
  08:30:00 STEP 2625 H=2625:00 +CPU= 23.934
  08:30:16 STEP 2626 H=2626:00 +CPU= 15.912
  08:30:32 STEP 2627 H=2627:00 +CPU= 15.468
  08:30:58 STEP 2628 H=2628:00 +CPU= 26.144
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002628
  08:31:14 STEP 2629 H=2629:00 +CPU= 15.563
  08:31:30 STEP 2630 H=2630:00 +CPU= 15.718
  08:31:53 STEP 2631 H=2631:00 +CPU= 23.768
  08:32:10 STEP 2632 H=2632:00 +CPU= 15.948
  08:32:25 STEP 2633 H=2633:00 +CPU= 15.490
  08:32:49 STEP 2634 H=2634:00 +CPU= 24.237
  08:33:05 STEP 2635 H=2635:00 +CPU= 15.625
  08:33:21 STEP 2636 H=2636:00 +CPU= 15.959
  08:33:45 STEP 2637 H=2637:00 +CPU= 23.820
  08:34:01 STEP 2638 H=2638:00 +CPU= 15.820
  08:34:17 STEP 2639 H=2639:00 +CPU= 16.000
  08:34:46 STEP 2640 H=2640:00 +CPU= 28.193
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_109.zip
Uploading the intermediate file: upload_file_109.zip
  08:35:01 STEP 2641 H=2641:00 +CPU= 15.256
  08:35:17 STEP 2642 H=2642:00 +CPU= 15.846
  08:35:41 STEP 2643 H=2643:00 +CPU= 23.801
  08:35:57 STEP 2644 H=2644:00 +CPU= 15.773
  08:36:12 STEP 2645 H=2645:00 +CPU= 15.679
  08:36:37 STEP 2646 H=2646:00 +CPU= 24.635
  08:36:53 STEP 2647 H=2647:00 +CPU= 15.374
  08:37:09 STEP 2648 H=2648:00 +CPU= 15.895
  08:37:33 STEP 2649 H=2649:00 +CPU= 23.795
  08:37:48 STEP 2650 H=2650:00 +CPU= 15.571
  08:38:04 STEP 2651 H=2651:00 +CPU= 15.369
  08:38:30 STEP 2652 H=2652:00 +CPU= 26.504
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002652
  08:38:46 STEP 2653 H=2653:00 +CPU= 15.591
  08:39:02 STEP 2654 H=2654:00 +CPU= 16.219
  08:39:27 STEP 2655 H=2655:00 +CPU= 24.365
  08:39:43 STEP 2656 H=2656:00 +CPU= 15.845
  08:39:58 STEP 2657 H=2657:00 +CPU= 15.722
  08:40:23 STEP 2658 H=2658:00 +CPU= 24.417
  08:40:38 STEP 2659 H=2659:00 +CPU= 15.325
  08:40:54 STEP 2660 H=2660:00 +CPU= 15.850
  08:41:18 STEP 2661 H=2661:00 +CPU= 23.836
  08:41:34 STEP 2662 H=2662:00 +CPU= 15.890
  08:41:50 STEP 2663 H=2663:00 +CPU= 16.067
  08:42:19 STEP 2664 H=2664:00 +CPU= 28.109
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_110.zip
Uploading the intermediate file: upload_file_110.zip
  08:42:34 STEP 2665 H=2665:00 +CPU= 15.038
  08:42:50 STEP 2666 H=2666:00 +CPU= 15.676
  08:43:13 STEP 2667 H=2667:00 +CPU= 23.552
  08:43:29 STEP 2668 H=2668:00 +CPU= 15.737
  08:43:45 STEP 2669 H=2669:00 +CPU= 15.351
  08:44:09 STEP 2670 H=2670:00 +CPU= 24.140
  08:44:24 STEP 2671 H=2671:00 +CPU= 15.265
  08:44:40 STEP 2672 H=2672:00 +CPU= 15.757
  08:45:04 STEP 2673 H=2673:00 +CPU= 23.926
  08:45:20 STEP 2674 H=2674:00 +CPU= 15.878
  08:45:36 STEP 2675 H=2675:00 +CPU= 15.591
  08:46:02 STEP 2676 H=2676:00 +CPU= 26.498
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002676
  08:46:18 STEP 2677 H=2677:00 +CPU= 15.585
  08:46:34 STEP 2678 H=2678:00 +CPU= 15.782
  08:46:58 STEP 2679 H=2679:00 +CPU= 23.861
  08:47:14 STEP 2680 H=2680:00 +CPU= 15.945
  08:47:29 STEP 2681 H=2681:00 +CPU= 15.452
  08:47:54 STEP 2682 H=2682:00 +CPU= 24.183
  08:48:09 STEP 2683 H=2683:00 +CPU= 15.383
  08:48:25 STEP 2684 H=2684:00 +CPU= 15.798
  08:48:49 STEP 2685 H=2685:00 +CPU= 23.631
  08:49:05 STEP 2686 H=2686:00 +CPU= 15.949
  08:49:21 STEP 2687 H=2687:00 +CPU= 15.824
  08:49:49 STEP 2688 H=2688:00 +CPU= 28.163
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_111.zip
Uploading the intermediate file: upload_file_111.zip
  08:50:05 STEP 2689 H=2689:00 +CPU= 15.348
  08:50:21 STEP 2690 H=2690:00 +CPU= 16.044
  08:50:45 STEP 2691 H=2691:00 +CPU= 23.851
  08:51:01 STEP 2692 H=2692:00 +CPU= 15.908
  08:51:16 STEP 2693 H=2693:00 +CPU= 15.505
  08:51:41 STEP 2694 H=2694:00 +CPU= 24.147
  08:51:56 STEP 2695 H=2695:00 +CPU= 15.393
  08:52:12 STEP 2696 H=2696:00 +CPU= 16.026
  08:52:36 STEP 2697 H=2697:00 +CPU= 23.894
  08:52:52 STEP 2698 H=2698:00 +CPU= 15.775
  08:53:08 STEP 2699 H=2699:00 +CPU= 15.426
  08:53:34 STEP 2700 H=2700:00 +CPU= 26.494
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002700
  08:53:50 STEP 2701 H=2701:00 +CPU= 15.527
  08:54:06 STEP 2702 H=2702:00 +CPU= 15.731
  08:54:30 STEP 2703 H=2703:00 +CPU= 24.022
  08:54:46 STEP 2704 H=2704:00 +CPU= 15.974
  08:55:01 STEP 2705 H=2705:00 +CPU= 15.543
  08:55:26 STEP 2706 H=2706:00 +CPU= 24.471
  08:55:42 STEP 2707 H=2707:00 +CPU= 15.379
  08:55:57 STEP 2708 H=2708:00 +CPU= 15.900
  08:56:22 STEP 2709 H=2709:00 +CPU= 24.004
  08:56:38 STEP 2710 H=2710:00 +CPU= 15.937
  08:56:54 STEP 2711 H=2711:00 +CPU= 16.114
  08:57:22 STEP 2712 H=2712:00 +CPU= 28.301
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_112.zip
Uploading the intermediate file: upload_file_112.zip
  08:57:38 STEP 2713 H=2713:00 +CPU= 15.669
  08:57:54 STEP 2714 H=2714:00 +CPU= 15.845
  08:58:18 STEP 2715 H=2715:00 +CPU= 24.272
  08:58:34 STEP 2716 H=2716:00 +CPU= 15.818
  08:58:50 STEP 2717 H=2717:00 +CPU= 15.620
  08:59:15 STEP 2718 H=2718:00 +CPU= 24.473
  08:59:30 STEP 2719 H=2719:00 +CPU= 15.555
  08:59:46 STEP 2720 H=2720:00 +CPU= 16.131
  09:00:11 STEP 2721 H=2721:00 +CPU= 24.104
  09:00:27 STEP 2722 H=2722:00 +CPU= 15.919
  09:00:42 STEP 2723 H=2723:00 +CPU= 15.503
  09:01:09 STEP 2724 H=2724:00 +CPU= 26.427
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002724
  09:01:25 STEP 2725 H=2725:00 +CPU= 15.511
  09:01:41 STEP 2726 H=2726:00 +CPU= 16.025
  09:02:05 STEP 2727 H=2727:00 +CPU= 23.992
  09:02:21 STEP 2728 H=2728:00 +CPU= 16.004
  09:02:37 STEP 2729 H=2729:00 +CPU= 15.583
  09:03:01 STEP 2730 H=2730:00 +CPU= 24.412
  09:03:17 STEP 2731 H=2731:00 +CPU= 15.520
  09:03:32 STEP 2732 H=2732:00 +CPU= 15.695
  09:03:56 STEP 2733 H=2733:00 +CPU= 23.952
  09:04:13 STEP 2734 H=2734:00 +CPU= 16.076
  09:04:29 STEP 2735 H=2735:00 +CPU= 16.023
  09:04:57 STEP 2736 H=2736:00 +CPU= 28.264
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_113.zip
Uploading the intermediate file: upload_file_113.zip
  09:05:13 STEP 2737 H=2737:00 +CPU= 15.370
  09:05:29 STEP 2738 H=2738:00 +CPU= 15.805
  09:05:53 STEP 2739 H=2739:00 +CPU= 23.815
  09:06:08 STEP 2740 H=2740:00 +CPU= 15.633
  09:06:24 STEP 2741 H=2741:00 +CPU= 15.607
  09:06:49 STEP 2742 H=2742:00 +CPU= 24.464
  09:07:04 STEP 2743 H=2743:00 +CPU= 15.651
  09:07:20 STEP 2744 H=2744:00 +CPU= 16.091
  09:07:45 STEP 2745 H=2745:00 +CPU= 24.075
  09:08:01 STEP 2746 H=2746:00 +CPU= 16.050
  09:08:16 STEP 2747 H=2747:00 +CPU= 15.479
  09:08:43 STEP 2748 H=2748:00 +CPU= 26.766
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002748
  09:08:59 STEP 2749 H=2749:00 +CPU= 15.513
  09:09:15 STEP 2750 H=2750:00 +CPU= 15.833
  09:09:39 STEP 2751 H=2751:00 +CPU= 23.761
  09:09:55 STEP 2752 H=2752:00 +CPU= 15.799
  09:10:10 STEP 2753 H=2753:00 +CPU= 15.561
  09:10:34 STEP 2754 H=2754:00 +CPU= 23.917
  09:10:50 STEP 2755 H=2755:00 +CPU= 15.582
  09:11:06 STEP 2756 H=2756:00 +CPU= 15.762
  09:11:30 STEP 2757 H=2757:00 +CPU= 23.757
  09:11:45 STEP 2758 H=2758:00 +CPU= 15.762
  09:12:01 STEP 2759 H=2759:00 +CPU= 15.731
  09:12:29 STEP 2760 H=2760:00 +CPU= 27.940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_114.zip
Uploading the intermediate file: upload_file_114.zip
  09:12:45 STEP 2761 H=2761:00 +CPU= 15.351
  09:13:01 STEP 2762 H=2762:00 +CPU= 15.627
  09:13:25 STEP 2763 H=2763:00 +CPU= 23.886
  09:13:41 STEP 2764 H=2764:00 +CPU= 15.879
  09:13:56 STEP 2765 H=2765:00 +CPU= 15.328
  09:14:21 STEP 2766 H=2766:00 +CPU= 24.443
  09:14:36 STEP 2767 H=2767:00 +CPU= 15.402
  09:14:52 STEP 2768 H=2768:00 +CPU= 15.675
  09:15:16 STEP 2769 H=2769:00 +CPU= 23.990
  09:15:32 STEP 2770 H=2770:00 +CPU= 15.928
  09:15:47 STEP 2771 H=2771:00 +CPU= 15.416
  09:16:14 STEP 2772 H=2772:00 +CPU= 26.541
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002772
  09:16:29 STEP 2773 H=2773:00 +CPU= 15.311
  09:16:45 STEP 2774 H=2774:00 +CPU= 15.736
  09:17:09 STEP 2775 H=2775:00 +CPU= 23.861
  09:17:25 STEP 2776 H=2776:00 +CPU= 15.770
  09:17:41 STEP 2777 H=2777:00 +CPU= 15.332
  09:18:05 STEP 2778 H=2778:00 +CPU= 24.270
  09:18:20 STEP 2779 H=2779:00 +CPU= 15.466
  09:18:36 STEP 2780 H=2780:00 +CPU= 15.708
  09:19:00 STEP 2781 H=2781:00 +CPU= 23.917
  09:19:16 STEP 2782 H=2782:00 +CPU= 15.792
  09:19:32 STEP 2783 H=2783:00 +CPU= 15.808
  09:20:01 STEP 2784 H=2784:00 +CPU= 28.288
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_115.zip
Uploading the intermediate file: upload_file_115.zip
  09:20:16 STEP 2785 H=2785:00 +CPU= 15.415
Uploading trickle at timestep: 10022400
  09:20:32 STEP 2786 H=2786:00 +CPU= 15.559
  09:20:56 STEP 2787 H=2787:00 +CPU= 23.873
  09:21:12 STEP 2788 H=2788:00 +CPU= 15.829
  09:21:27 STEP 2789 H=2789:00 +CPU= 15.444
  09:21:52 STEP 2790 H=2790:00 +CPU= 24.289
  09:22:07 STEP 2791 H=2791:00 +CPU= 15.277
  09:22:23 STEP 2792 H=2792:00 +CPU= 15.790
  09:22:47 STEP 2793 H=2793:00 +CPU= 23.792
  09:23:03 STEP 2794 H=2794:00 +CPU= 15.972
  09:23:18 STEP 2795 H=2795:00 +CPU= 15.361
  09:23:45 STEP 2796 H=2796:00 +CPU= 26.740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002796
  09:24:01 STEP 2797 H=2797:00 +CPU= 15.309
  09:24:16 STEP 2798 H=2798:00 +CPU= 15.875
  09:24:40 STEP 2799 H=2799:00 +CPU= 23.750
  09:24:56 STEP 2800 H=2800:00 +CPU= 15.669
  09:25:12 STEP 2801 H=2801:00 +CPU= 15.489
  09:25:36 STEP 2802 H=2802:00 +CPU= 24.260
  09:25:52 STEP 2803 H=2803:00 +CPU= 15.363
  09:26:07 STEP 2804 H=2804:00 +CPU= 15.623
  09:26:31 STEP 2805 H=2805:00 +CPU= 23.787
  09:26:47 STEP 2806 H=2806:00 +CPU= 15.741
  09:27:03 STEP 2807 H=2807:00 +CPU= 15.629
  09:27:31 STEP 2808 H=2808:00 +CPU= 28.288
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_116.zip
Uploading the intermediate file: upload_file_116.zip
  09:27:47 STEP 2809 H=2809:00 +CPU= 15.363
  09:28:02 STEP 2810 H=2810:00 +CPU= 15.608
  09:28:26 STEP 2811 H=2811:00 +CPU= 23.815
  09:28:42 STEP 2812 H=2812:00 +CPU= 15.847
  09:28:58 STEP 2813 H=2813:00 +CPU= 15.422
  09:29:22 STEP 2814 H=2814:00 +CPU= 24.302
  09:29:38 STEP 2815 H=2815:00 +CPU= 15.390
  09:29:53 STEP 2816 H=2816:00 +CPU= 15.780
  09:30:17 STEP 2817 H=2817:00 +CPU= 23.858
  09:30:33 STEP 2818 H=2818:00 +CPU= 15.631
  09:30:49 STEP 2819 H=2819:00 +CPU= 15.373
  09:31:15 STEP 2820 H=2820:00 +CPU= 26.227
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002820
  09:31:30 STEP 2821 H=2821:00 +CPU= 15.184
  09:31:46 STEP 2822 H=2822:00 +CPU= 15.723
  09:32:10 STEP 2823 H=2823:00 +CPU= 23.699
  09:32:26 STEP 2824 H=2824:00 +CPU= 15.727
  09:32:41 STEP 2825 H=2825:00 +CPU= 15.170
  09:33:05 STEP 2826 H=2826:00 +CPU= 23.919
  09:33:20 STEP 2827 H=2827:00 +CPU= 15.399
  09:33:36 STEP 2828 H=2828:00 +CPU= 15.783
  09:34:00 STEP 2829 H=2829:00 +CPU= 23.526
  09:34:16 STEP 2830 H=2830:00 +CPU= 15.823
  09:34:32 STEP 2831 H=2831:00 +CPU= 15.664
  09:35:00 STEP 2832 H=2832:00 +CPU= 27.944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_117.zip
Uploading the intermediate file: upload_file_117.zip
  09:35:15 STEP 2833 H=2833:00 +CPU= 15.427
  09:35:31 STEP 2834 H=2834:00 +CPU= 15.629
  09:35:55 STEP 2835 H=2835:00 +CPU= 23.550
  09:36:11 STEP 2836 H=2836:00 +CPU= 15.940
  09:36:26 STEP 2837 H=2837:00 +CPU= 15.293
  09:36:50 STEP 2838 H=2838:00 +CPU= 23.963
  09:37:05 STEP 2839 H=2839:00 +CPU= 15.147
  09:37:21 STEP 2840 H=2840:00 +CPU= 15.802
  09:37:45 STEP 2841 H=2841:00 +CPU= 23.723
  09:38:01 STEP 2842 H=2842:00 +CPU= 15.770
  09:38:16 STEP 2843 H=2843:00 +CPU= 15.431
  09:38:43 STEP 2844 H=2844:00 +CPU= 26.745
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002844
  09:38:59 STEP 2845 H=2845:00 +CPU= 15.289
  09:39:15 STEP 2846 H=2846:00 +CPU= 15.888
  09:39:39 STEP 2847 H=2847:00 +CPU= 23.868
  09:39:54 STEP 2848 H=2848:00 +CPU= 15.637
  09:40:10 STEP 2849 H=2849:00 +CPU= 15.581
  09:40:34 STEP 2850 H=2850:00 +CPU= 24.251
  09:40:50 STEP 2851 H=2851:00 +CPU= 15.354
  09:41:06 STEP 2852 H=2852:00 +CPU= 15.781
  09:41:29 STEP 2853 H=2853:00 +CPU= 23.506
  09:41:45 STEP 2854 H=2854:00 +CPU= 15.693
  09:42:01 STEP 2855 H=2855:00 +CPU= 15.744
  09:42:29 STEP 2856 H=2856:00 +CPU= 27.938
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_118.zip
Uploading the intermediate file: upload_file_118.zip
  09:42:44 STEP 2857 H=2857:00 +CPU= 15.148
  09:43:00 STEP 2858 H=2858:00 +CPU= 15.498
  09:43:24 STEP 2859 H=2859:00 +CPU= 23.780
  09:43:39 STEP 2860 H=2860:00 +CPU= 15.604
  09:43:55 STEP 2861 H=2861:00 +CPU= 15.285
  09:44:19 STEP 2862 H=2862:00 +CPU= 24.381
  09:44:35 STEP 2863 H=2863:00 +CPU= 15.405
  09:44:51 STEP 2864 H=2864:00 +CPU= 15.722
  09:45:15 STEP 2865 H=2865:00 +CPU= 23.791
  09:45:31 STEP 2866 H=2866:00 +CPU= 15.941
  09:45:46 STEP 2867 H=2867:00 +CPU= 15.505
  09:46:13 STEP 2868 H=2868:00 +CPU= 26.499
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002868
  09:46:28 STEP 2869 H=2869:00 +CPU= 15.292
  09:46:44 STEP 2870 H=2870:00 +CPU= 15.617
  09:47:08 STEP 2871 H=2871:00 +CPU= 23.634
  09:47:24 STEP 2872 H=2872:00 +CPU= 15.801
  09:47:39 STEP 2873 H=2873:00 +CPU= 15.478
  09:48:03 STEP 2874 H=2874:00 +CPU= 24.261
  09:48:19 STEP 2875 H=2875:00 +CPU= 15.433
  09:48:35 STEP 2876 H=2876:00 +CPU= 15.745
  09:48:58 STEP 2877 H=2877:00 +CPU= 23.532
  09:49:14 STEP 2878 H=2878:00 +CPU= 15.820
  09:49:30 STEP 2879 H=2879:00 +CPU= 15.750
  09:49:58 STEP 2880 H=2880:00 +CPU= 28.037
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_119.zip
Uploading the intermediate file: upload_file_119.zip
  09:50:14 STEP 2881 H=2881:00 +CPU= 15.384
  09:50:30 STEP 2882 H=2882:00 +CPU= 15.832
  09:50:54 STEP 2883 H=2883:00 +CPU= 23.719
  09:51:09 STEP 2884 H=2884:00 +CPU= 15.763
  09:51:25 STEP 2885 H=2885:00 +CPU= 15.311
  09:51:49 STEP 2886 H=2886:00 +CPU= 24.132
  09:52:05 STEP 2887 H=2887:00 +CPU= 15.584
  09:52:21 STEP 2888 H=2888:00 +CPU= 15.780
  09:52:45 STEP 2889 H=2889:00 +CPU= 24.206
  09:53:01 STEP 2890 H=2890:00 +CPU= 15.801
  09:53:16 STEP 2891 H=2891:00 +CPU= 15.311
  09:53:43 STEP 2892 H=2892:00 +CPU= 26.417
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002892
  09:53:58 STEP 2893 H=2893:00 +CPU= 15.482
  09:54:14 STEP 2894 H=2894:00 +CPU= 15.777
  09:54:38 STEP 2895 H=2895:00 +CPU= 23.821
  09:54:54 STEP 2896 H=2896:00 +CPU= 16.030
  09:55:10 STEP 2897 H=2897:00 +CPU= 15.500
  09:55:34 STEP 2898 H=2898:00 +CPU= 24.152
  09:55:50 STEP 2899 H=2899:00 +CPU= 15.393
  09:56:06 STEP 2900 H=2900:00 +CPU= 15.779
  09:56:29 STEP 2901 H=2901:00 +CPU= 23.822
  09:56:45 STEP 2902 H=2902:00 +CPU= 15.902
  09:57:02 STEP 2903 H=2903:00 +CPU= 16.045
  09:57:30 STEP 2904 H=2904:00 +CPU= 28.318
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_120.zip
Uploading the intermediate file: upload_file_120.zip
  09:57:46 STEP 2905 H=2905:00 +CPU= 15.410
  09:58:01 STEP 2906 H=2906:00 +CPU= 15.717
  09:58:25 STEP 2907 H=2907:00 +CPU= 23.722
  09:58:41 STEP 2908 H=2908:00 +CPU= 15.946
  09:58:57 STEP 2909 H=2909:00 +CPU= 15.571
  09:59:22 STEP 2910 H=2910:00 +CPU= 24.472
  09:59:37 STEP 2911 H=2911:00 +CPU= 15.717
  09:59:53 STEP 2912 H=2912:00 +CPU= 15.930
  10:00:18 STEP 2913 H=2913:00 +CPU= 24.202
  10:00:34 STEP 2914 H=2914:00 +CPU= 15.935
  10:00:49 STEP 2915 H=2915:00 +CPU= 15.594
  10:01:16 STEP 2916 H=2916:00 +CPU= 26.663
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002916
  10:01:32 STEP 2917 H=2917:00 +CPU= 15.692
  10:01:48 STEP 2918 H=2918:00 +CPU= 15.999
  10:02:12 STEP 2919 H=2919:00 +CPU= 24.016
  10:02:28 STEP 2920 H=2920:00 +CPU= 15.982
  10:02:44 STEP 2921 H=2921:00 +CPU= 15.528
  10:03:08 STEP 2922 H=2922:00 +CPU= 24.348
  10:03:24 STEP 2923 H=2923:00 +CPU= 15.482
  10:03:40 STEP 2924 H=2924:00 +CPU= 15.868
  10:04:04 STEP 2925 H=2925:00 +CPU= 24.030
  10:04:20 STEP 2926 H=2926:00 +CPU= 16.041
  10:04:36 STEP 2927 H=2927:00 +CPU= 16.085
  10:05:05 STEP 2928 H=2928:00 +CPU= 28.595
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_121.zip
Uploading the intermediate file: upload_file_121.zip
  10:05:21 STEP 2929 H=2929:00 +CPU= 15.581
  10:05:37 STEP 2930 H=2930:00 +CPU= 16.083
  10:06:01 STEP 2931 H=2931:00 +CPU= 24.154
  10:06:18 STEP 2932 H=2932:00 +CPU= 16.161
  10:06:34 STEP 2933 H=2933:00 +CPU= 15.810
  10:06:58 STEP 2934 H=2934:00 +CPU= 24.624
  10:07:14 STEP 2935 H=2935:00 +CPU= 15.815
  10:07:30 STEP 2936 H=2936:00 +CPU= 16.118
  10:07:55 STEP 2937 H=2937:00 +CPU= 24.054
  10:08:11 STEP 2938 H=2938:00 +CPU= 16.138
  10:08:27 STEP 2939 H=2939:00 +CPU= 15.792
  10:08:54 STEP 2940 H=2940:00 +CPU= 26.643
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002940
  10:09:09 STEP 2941 H=2941:00 +CPU= 15.434
  10:09:25 STEP 2942 H=2942:00 +CPU= 16.096
  10:09:49 STEP 2943 H=2943:00 +CPU= 23.873
  10:10:05 STEP 2944 H=2944:00 +CPU= 15.903
  10:10:21 STEP 2945 H=2945:00 +CPU= 15.524
  10:10:46 STEP 2946 H=2946:00 +CPU= 24.548
  10:11:01 STEP 2947 H=2947:00 +CPU= 15.604
  10:11:18 STEP 2948 H=2948:00 +CPU= 16.101
  10:11:42 STEP 2949 H=2949:00 +CPU= 24.182
  10:11:58 STEP 2950 H=2950:00 +CPU= 16.112
  10:12:14 STEP 2951 H=2951:00 +CPU= 16.196
  10:12:42 STEP 2952 H=2952:00 +CPU= 27.671
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172670/ICMGGhq0f+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0026_1987050100_123_956_12172670_1_r1854883992_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
10:16:05 (2153): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:19:01 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 10,623,600 55,495 0.0052
09 May 2023 16:18:58 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 10,022,400 52,356 0.0052
09 May 2023 16:18:55 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 9,158,400 47,844 0.0052
09 May 2023 16:17:36 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 8,294,400 43,363 0.0052
09 May 2023 16:17:32 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 7,430,400 38,892 0.0052
09 May 2023 16:17:29 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 6,566,400 34,401 0.0052
09 May 2023 16:17:24 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 5,702,400 29,910 0.0052
09 May 2023 16:17:21 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 4,838,400 25,420 0.0053
09 May 2023 16:17:17 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 3,974,400 20,950 0.0053
09 May 2023 16:17:13 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 3,110,400 16,460 0.0053
09 May 2023 16:17:09 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 2,246,400 11,970 0.0053
09 May 2023 16:17:05 1513601 22296420 oifs_43r3_ps_0026_1987050100_123_956_12172670_1 1,382,400 7,491 0.0054


©2024 climateprediction.net