climateprediction.net home page
Task 22442581

Task 22442581

Name oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0
Workunit 12289006
Created 12 Jun 2024, 20:57:29 UTC
Sent 14 Jun 2024, 8:07:44 UTC
Report deadline 13 Aug 2024, 8:07:44 UTC
Received 17 Jun 2024, 8:42:23 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1539347
Run time 21 hours 48 min 30 sec
CPU time 21 hours 40 min 43 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 5.61 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.97 MB
Peak swap size 5,981.08 MB
Peak disk usage 1,346.10 MB
Stderr
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
 380:30 +CPU= 38.755
  14:21:27 STEP 1523 H= 380:45 +CPU= 38.796
  14:22:16 STEP 1524 H= 381:00 +CPU= 48.733
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001524
  14:22:55 STEP 1525 H= 381:15 +CPU= 38.751
  14:23:34 STEP 1526 H= 381:30 +CPU= 38.777
  14:24:12 STEP 1527 H= 381:45 +CPU= 38.760
  14:24:52 STEP 1528 H= 382:00 +CPU= 39.330
  14:25:31 STEP 1529 H= 382:15 +CPU= 38.757
  14:26:10 STEP 1530 H= 382:30 +CPU= 38.832
  14:26:49 STEP 1531 H= 382:45 +CPU= 38.843
  14:27:28 STEP 1532 H= 383:00 +CPU= 39.348
  14:28:07 STEP 1533 H= 383:15 +CPU= 38.814
  14:28:46 STEP 1534 H= 383:30 +CPU= 38.758
  14:29:25 STEP 1535 H= 383:45 +CPU= 38.710
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2603MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 2679 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        DYNFPOS 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         STEPO_FPOS 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          IOFPOS 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           WRPLFP 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            WROUTGPGB 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             IOSTREAM_MIX:IO_PUT 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              IOSTREAM_MIX:GRID_OUT 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               IOSTREAM_MIX:WRITE_RECORD 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                IOSTREAM_MIX:GRIB_ENCODE 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 >OMP-IOSTREAM_MIX: GRIB_AP(1484) 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  GRIB_CODE_MESSAGE 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                   GRIB_UTILS_MOD:GRIB_SET_TIME 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150615:1718575575:65091.987] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                    GRIB_API:IGRIB_SET_INT 
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150616:1718575576:65092.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:104058:104058] [20240616:150616:1718575576:65092.987] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001B373BD  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001B37224  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001B00B10  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001AE53C0  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001ADC69D  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000A6C505  grib_api_interfac         230  grib_api_interface.F90
oifs_43r3_model.e  0000000000A71175  grib_utils_mod_mp         347  grib_utils_mod.F90
oifs_43r3_model.e  0000000000A99C95  grib_code_message         198  grib_code_message.F90
oifs_43r3_model.e  0000000000A4AAA3  iostream_mix_mp_w        3122  iostream_mix.F90
oifs_43r3_model.e  0000000000A41E63  iostream_mix_mp_g        2087  iostream_mix.F90
oifs_43r3_model.e  0000000000A3CD09  iostream_mix_mp_i        1311  iostream_mix.F90
oifs_43r3_model.e  0000000000500FAC  wroutgpgb_                234  wroutgpgb.F90
oifs_43r3_model.e  00000000005034D2  wrplfp_                   194  wrplfp.F90
oifs_43r3_model.e  00000000004FCCDD  iofpos_                    95  iofpos.F90
oifs_43r3_model.e  00000000004D7ABF  stepo_fpos_               556  stepo_fpos.F90
oifs_43r3_model.e  00000000004CFC43  dynfpos_                  170  dynfpos.F90
oifs_43r3_model.e  00000000004CCCC8  cnt4_                     947  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0iv
(argv4) batchid: 1018
(argv5) wuid: 12289006
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_450703e9b3275bad16ab717fa30724be to: /var/lib/boinc-client/slots/1/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006.zip
ic_ancil_file: ic_ancil_12289006
ifsdata_file: ifsdata_12289006
climate_data_file: clim_data_12289006
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/1/ic_ancil_12289006.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12289006.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_58a46c834126d1de30bca85b404b1dcb to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12289006.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12289006.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12289006.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12289006.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12289006.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12289006.xml
-- Model is restarting --
Adjusting last_iter, 1536, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12289006.xml
last_cpu_time: 62691
upload_file_number: 15
last_iter: 1537
last_upload: 1296000
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239
The child process has been launched with process id: 3611
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:3611:3611] [20240616:212159:1718598119:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239_15.zip
Uploading the intermediate file: upload_file_15.zip
  21:22:46 STEP 1536 H= 384:00 +CPU= 39.999
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001536
  21:23:25 STEP 1537 H= 384:15 +CPU= 38.690
  21:24:03 STEP 1538 H= 384:30 +CPU= 38.599
  21:24:42 STEP 1539 H= 384:45 +CPU= 38.710
  21:25:22 STEP 1540 H= 385:00 +CPU= 39.259
  21:26:00 STEP 1541 H= 385:15 +CPU= 38.696
  21:26:39 STEP 1542 H= 385:30 +CPU= 38.737
  21:27:18 STEP 1543 H= 385:45 +CPU= 38.707
  21:27:57 STEP 1544 H= 386:00 +CPU= 39.270
  21:28:36 STEP 1545 H= 386:15 +CPU= 38.682
  21:29:15 STEP 1546 H= 386:30 +CPU= 38.649
  21:29:54 STEP 1547 H= 386:45 +CPU= 38.754
  21:30:43 STEP 1548 H= 387:00 +CPU= 49.050
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001548
  21:31:22 STEP 1549 H= 387:15 +CPU= 39.101
  21:32:01 STEP 1550 H= 387:30 +CPU= 38.964
  21:32:40 STEP 1551 H= 387:45 +CPU= 38.752
  21:33:20 STEP 1552 H= 388:00 +CPU= 39.395
  21:33:59 STEP 1553 H= 388:15 +CPU= 38.859
  21:34:38 STEP 1554 H= 388:30 +CPU= 38.823
  21:35:17 STEP 1555 H= 388:45 +CPU= 38.753
  21:35:56 STEP 1556 H= 389:00 +CPU= 39.313
  21:36:35 STEP 1557 H= 389:15 +CPU= 38.823
  21:37:14 STEP 1558 H= 389:30 +CPU= 38.771
  21:37:53 STEP 1559 H= 389:45 +CPU= 38.780
  21:38:42 STEP 1560 H= 390:00 +CPU= 49.199
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001560
  21:39:21 STEP 1561 H= 390:15 +CPU= 38.963
  21:40:00 STEP 1562 H= 390:30 +CPU= 38.888
  21:40:39 STEP 1563 H= 390:45 +CPU= 38.915
  21:41:19 STEP 1564 H= 391:00 +CPU= 39.467
  21:41:58 STEP 1565 H= 391:15 +CPU= 38.870
  21:42:37 STEP 1566 H= 391:30 +CPU= 38.992
  21:43:16 STEP 1567 H= 391:45 +CPU= 38.957
  21:43:55 STEP 1568 H= 392:00 +CPU= 39.538
  21:44:35 STEP 1569 H= 392:15 +CPU= 38.996
  21:45:14 STEP 1570 H= 392:30 +CPU= 38.920
  21:45:53 STEP 1571 H= 392:45 +CPU= 38.963
  21:46:42 STEP 1572 H= 393:00 +CPU= 49.083
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001572
  21:47:21 STEP 1573 H= 393:15 +CPU= 38.918
  21:48:00 STEP 1574 H= 393:30 +CPU= 38.917
  21:48:39 STEP 1575 H= 393:45 +CPU= 38.866
  21:49:18 STEP 1576 H= 394:00 +CPU= 39.506
  21:49:57 STEP 1577 H= 394:15 +CPU= 38.859
  21:50:36 STEP 1578 H= 394:30 +CPU= 38.906
  21:51:15 STEP 1579 H= 394:45 +CPU= 38.838
  21:51:55 STEP 1580 H= 395:00 +CPU= 39.389
  21:52:34 STEP 1581 H= 395:15 +CPU= 38.911
  21:53:13 STEP 1582 H= 395:30 +CPU= 38.945
  21:53:52 STEP 1583 H= 395:45 +CPU= 38.935
  21:54:46 STEP 1584 H= 396:00 +CPU= 53.859
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001584
  21:55:25 STEP 1585 H= 396:15 +CPU= 38.748
  21:56:04 STEP 1586 H= 396:30 +CPU= 38.703
  21:56:42 STEP 1587 H= 396:45 +CPU= 38.688
  21:57:22 STEP 1588 H= 397:00 +CPU= 39.238
  21:58:00 STEP 1589 H= 397:15 +CPU= 38.627
  21:58:39 STEP 1590 H= 397:30 +CPU= 38.712
  21:59:18 STEP 1591 H= 397:45 +CPU= 38.656
  21:59:57 STEP 1592 H= 398:00 +CPU= 39.279
  22:00:36 STEP 1593 H= 398:15 +CPU= 38.674
  22:01:15 STEP 1594 H= 398:30 +CPU= 38.677
  22:01:54 STEP 1595 H= 398:45 +CPU= 38.687
  22:02:43 STEP 1596 H= 399:00 +CPU= 48.878
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001596
  22:03:22 STEP 1597 H= 399:15 +CPU= 38.865
  22:04:00 STEP 1598 H= 399:30 +CPU= 38.619
  22:04:39 STEP 1599 H= 399:45 +CPU= 38.648
  22:05:19 STEP 1600 H= 400:00 +CPU= 39.358
  22:05:57 STEP 1601 H= 400:15 +CPU= 38.751
  22:06:36 STEP 1602 H= 400:30 +CPU= 38.732
  22:07:15 STEP 1603 H= 400:45 +CPU= 38.674
  22:07:54 STEP 1604 H= 401:00 +CPU= 39.302
  22:08:33 STEP 1605 H= 401:15 +CPU= 38.709
  22:09:12 STEP 1606 H= 401:30 +CPU= 38.775
  22:09:51 STEP 1607 H= 401:45 +CPU= 38.727
  22:10:40 STEP 1608 H= 402:00 +CPU= 49.067
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001608
  22:11:19 STEP 1609 H= 402:15 +CPU= 38.768
  22:11:58 STEP 1610 H= 402:30 +CPU= 38.781
  22:12:37 STEP 1611 H= 402:45 +CPU= 38.776
  22:13:16 STEP 1612 H= 403:00 +CPU= 39.286
  22:13:55 STEP 1613 H= 403:15 +CPU= 38.832
  22:14:34 STEP 1614 H= 403:30 +CPU= 38.839
  22:15:13 STEP 1615 H= 403:45 +CPU= 38.796
  22:15:52 STEP 1616 H= 404:00 +CPU= 39.390
  22:16:31 STEP 1617 H= 404:15 +CPU= 38.844
  22:17:10 STEP 1618 H= 404:30 +CPU= 38.855
  22:17:49 STEP 1619 H= 404:45 +CPU= 38.890
  22:18:38 STEP 1620 H= 405:00 +CPU= 49.031
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001620
  22:19:17 STEP 1621 H= 405:15 +CPU= 38.919
  22:19:56 STEP 1622 H= 405:30 +CPU= 38.719
  22:20:35 STEP 1623 H= 405:45 +CPU= 38.758
  22:21:15 STEP 1624 H= 406:00 +CPU= 39.263
  22:21:53 STEP 1625 H= 406:15 +CPU= 38.658
  22:22:32 STEP 1626 H= 406:30 +CPU= 38.665
  22:23:11 STEP 1627 H= 406:45 +CPU= 38.755
  22:23:50 STEP 1628 H= 407:00 +CPU= 39.352
  22:24:29 STEP 1629 H= 407:15 +CPU= 38.761
  22:25:08 STEP 1630 H= 407:30 +CPU= 38.770
  22:25:47 STEP 1631 H= 407:45 +CPU= 38.703
  22:26:41 STEP 1632 H= 408:00 +CPU= 54.052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001632
  22:27:20 STEP 1633 H= 408:15 +CPU= 38.777
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239_16.zip
Uploading the intermediate file: upload_file_16.zip
  22:27:59 STEP 1634 H= 408:30 +CPU= 38.754
  22:28:38 STEP 1635 H= 408:45 +CPU= 38.678
  22:29:17 STEP 1636 H= 409:00 +CPU= 39.304
  22:29:56 STEP 1637 H= 409:15 +CPU= 38.654
  22:30:35 STEP 1638 H= 409:30 +CPU= 38.809
  22:31:14 STEP 1639 H= 409:45 +CPU= 38.803
  22:31:53 STEP 1640 H= 410:00 +CPU= 39.303
  22:32:32 STEP 1641 H= 410:15 +CPU= 38.733
  22:33:11 STEP 1642 H= 410:30 +CPU= 38.808
  22:33:50 STEP 1643 H= 410:45 +CPU= 38.734
  22:34:39 STEP 1644 H= 411:00 +CPU= 49.059
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001644
  22:35:18 STEP 1645 H= 411:15 +CPU= 38.685
  22:35:57 STEP 1646 H= 411:30 +CPU= 38.727
  22:36:36 STEP 1647 H= 411:45 +CPU= 38.734
  22:37:15 STEP 1648 H= 412:00 +CPU= 39.398
  22:37:54 STEP 1649 H= 412:15 +CPU= 38.792
  22:38:33 STEP 1650 H= 412:30 +CPU= 38.663
  22:39:12 STEP 1651 H= 412:45 +CPU= 38.758
  22:39:51 STEP 1652 H= 413:00 +CPU= 39.241
  22:40:30 STEP 1653 H= 413:15 +CPU= 38.730
  22:41:09 STEP 1654 H= 413:30 +CPU= 38.814
  22:41:48 STEP 1655 H= 413:45 +CPU= 38.657
  22:42:37 STEP 1656 H= 414:00 +CPU= 48.976
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001656
  22:43:16 STEP 1657 H= 414:15 +CPU= 38.748
  22:43:55 STEP 1658 H= 414:30 +CPU= 38.855
  22:44:34 STEP 1659 H= 414:45 +CPU= 38.877
  22:45:13 STEP 1660 H= 415:00 +CPU= 39.399
  22:45:52 STEP 1661 H= 415:15 +CPU= 38.850
  22:46:31 STEP 1662 H= 415:30 +CPU= 38.871
  22:47:10 STEP 1663 H= 415:45 +CPU= 38.850
  22:47:50 STEP 1664 H= 416:00 +CPU= 39.393
  22:48:29 STEP 1665 H= 416:15 +CPU= 38.846
  22:49:08 STEP 1666 H= 416:30 +CPU= 38.942
  22:49:47 STEP 1667 H= 416:45 +CPU= 38.923
  22:50:36 STEP 1668 H= 417:00 +CPU= 49.174
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001668
  22:51:15 STEP 1669 H= 417:15 +CPU= 38.791
  22:51:54 STEP 1670 H= 417:30 +CPU= 38.680
  22:52:33 STEP 1671 H= 417:45 +CPU= 38.794
  22:53:12 STEP 1672 H= 418:00 +CPU= 39.308
  22:53:51 STEP 1673 H= 418:15 +CPU= 38.760
  22:54:30 STEP 1674 H= 418:30 +CPU= 38.801
  22:55:09 STEP 1675 H= 418:45 +CPU= 38.758
  22:55:48 STEP 1676 H= 419:00 +CPU= 39.406
  22:56:27 STEP 1677 H= 419:15 +CPU= 38.699
  22:57:06 STEP 1678 H= 419:30 +CPU= 38.723
  22:57:45 STEP 1679 H= 419:45 +CPU= 38.830
  22:58:39 STEP 1680 H= 420:00 +CPU= 54.017
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001680
  22:59:18 STEP 1681 H= 420:15 +CPU= 38.847
  22:59:57 STEP 1682 H= 420:30 +CPU= 38.912
  23:00:36 STEP 1683 H= 420:45 +CPU= 38.946
  23:01:16 STEP 1684 H= 421:00 +CPU= 39.448
  23:01:55 STEP 1685 H= 421:15 +CPU= 38.855
  23:02:34 STEP 1686 H= 421:30 +CPU= 38.839
  23:03:13 STEP 1687 H= 421:45 +CPU= 38.936
  23:03:53 STEP 1688 H= 422:00 +CPU= 39.455
  23:04:31 STEP 1689 H= 422:15 +CPU= 38.821
  23:05:10 STEP 1690 H= 422:30 +CPU= 38.819
  23:05:49 STEP 1691 H= 422:45 +CPU= 38.834
  23:06:38 STEP 1692 H= 423:00 +CPU= 48.906
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001692
  23:07:17 STEP 1693 H= 423:15 +CPU= 38.865
  23:07:56 STEP 1694 H= 423:30 +CPU= 38.808
  23:08:35 STEP 1695 H= 423:45 +CPU= 38.820
  23:09:15 STEP 1696 H= 424:00 +CPU= 39.427
  23:09:54 STEP 1697 H= 424:15 +CPU= 38.767
  23:10:33 STEP 1698 H= 424:30 +CPU= 38.870
  23:11:12 STEP 1699 H= 424:45 +CPU= 38.895
  23:11:51 STEP 1700 H= 425:00 +CPU= 39.450
  23:12:30 STEP 1701 H= 425:15 +CPU= 38.818
  23:13:09 STEP 1702 H= 425:30 +CPU= 38.818
  23:13:48 STEP 1703 H= 425:45 +CPU= 38.855
  23:14:37 STEP 1704 H= 426:00 +CPU= 49.093
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001704
  23:15:17 STEP 1705 H= 426:15 +CPU= 39.035
  23:15:56 STEP 1706 H= 426:30 +CPU= 38.906
  23:16:35 STEP 1707 H= 426:45 +CPU= 38.904
  23:17:14 STEP 1708 H= 427:00 +CPU= 39.467
  23:17:53 STEP 1709 H= 427:15 +CPU= 38.873
  23:18:32 STEP 1710 H= 427:30 +CPU= 38.859
  23:19:11 STEP 1711 H= 427:45 +CPU= 38.997
  23:19:51 STEP 1712 H= 428:00 +CPU= 39.428
  23:20:30 STEP 1713 H= 428:15 +CPU= 38.839
  23:21:09 STEP 1714 H= 428:30 +CPU= 38.933
  23:21:48 STEP 1715 H= 428:45 +CPU= 38.883
  23:22:37 STEP 1716 H= 429:00 +CPU= 49.033
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001716
  23:23:16 STEP 1717 H= 429:15 +CPU= 38.917
  23:23:55 STEP 1718 H= 429:30 +CPU= 38.870
  23:24:34 STEP 1719 H= 429:45 +CPU= 38.852
  23:25:14 STEP 1720 H= 430:00 +CPU= 39.485
  23:25:53 STEP 1721 H= 430:15 +CPU= 38.896
  23:26:32 STEP 1722 H= 430:30 +CPU= 38.874
  23:27:11 STEP 1723 H= 430:45 +CPU= 38.816
  23:27:50 STEP 1724 H= 431:00 +CPU= 39.404
  23:28:29 STEP 1725 H= 431:15 +CPU= 38.880
  23:29:08 STEP 1726 H= 431:30 +CPU= 38.883
  23:29:47 STEP 1727 H= 431:45 +CPU= 38.936
  23:30:41 STEP 1728 H= 432:00 +CPU= 54.138
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001728
  23:31:20 STEP 1729 H= 432:15 +CPU= 38.777
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001728
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239_17.zip
Uploading the intermediate file: upload_file_17.zip
  23:31:59 STEP 1730 H= 432:30 +CPU= 38.860
  23:32:38 STEP 1731 H= 432:45 +CPU= 38.904
  23:33:18 STEP 1732 H= 433:00 +CPU= 39.344
  23:33:57 STEP 1733 H= 433:15 +CPU= 38.714
  23:34:35 STEP 1734 H= 433:30 +CPU= 38.842
  23:35:14 STEP 1735 H= 433:45 +CPU= 38.835
  23:35:54 STEP 1736 H= 434:00 +CPU= 39.438
  23:36:33 STEP 1737 H= 434:15 +CPU= 38.801
  23:37:12 STEP 1738 H= 434:30 +CPU= 38.826
  23:37:51 STEP 1739 H= 434:45 +CPU= 38.823
  23:38:40 STEP 1740 H= 435:00 +CPU= 49.124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001740
  23:39:19 STEP 1741 H= 435:15 +CPU= 38.843
  23:39:58 STEP 1742 H= 435:30 +CPU= 38.755
  23:40:37 STEP 1743 H= 435:45 +CPU= 38.886
  23:41:16 STEP 1744 H= 436:00 +CPU= 39.447
  23:41:55 STEP 1745 H= 436:15 +CPU= 38.852
  23:42:34 STEP 1746 H= 436:30 +CPU= 38.830
  23:43:13 STEP 1747 H= 436:45 +CPU= 38.917
  23:43:53 STEP 1748 H= 437:00 +CPU= 39.394
  23:44:32 STEP 1749 H= 437:15 +CPU= 38.825
  23:45:11 STEP 1750 H= 437:30 +CPU= 38.919
  23:45:50 STEP 1751 H= 437:45 +CPU= 38.822
  23:46:39 STEP 1752 H= 438:00 +CPU= 49.216
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001752
  23:47:18 STEP 1753 H= 438:15 +CPU= 38.915
  23:47:57 STEP 1754 H= 438:30 +CPU= 38.898
  23:48:36 STEP 1755 H= 438:45 +CPU= 38.924
  23:49:16 STEP 1756 H= 439:00 +CPU= 39.459
  23:49:55 STEP 1757 H= 439:15 +CPU= 38.929
  23:50:34 STEP 1758 H= 439:30 +CPU= 38.912
  23:51:13 STEP 1759 H= 439:45 +CPU= 38.969
  23:51:53 STEP 1760 H= 440:00 +CPU= 39.424
  23:52:32 STEP 1761 H= 440:15 +CPU= 38.861
  23:53:11 STEP 1762 H= 440:30 +CPU= 38.874
  23:53:50 STEP 1763 H= 440:45 +CPU= 38.807
  23:54:39 STEP 1764 H= 441:00 +CPU= 49.124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001764
  23:55:18 STEP 1765 H= 441:15 +CPU= 38.790
  23:55:57 STEP 1766 H= 441:30 +CPU= 38.771
  23:56:36 STEP 1767 H= 441:45 +CPU= 38.845
  23:57:15 STEP 1768 H= 442:00 +CPU= 39.390
  23:57:54 STEP 1769 H= 442:15 +CPU= 38.825
  23:58:33 STEP 1770 H= 442:30 +CPU= 38.825
  23:59:12 STEP 1771 H= 442:45 +CPU= 38.888
  23:59:51 STEP 1772 H= 443:00 +CPU= 39.378
  00:00:30 STEP 1773 H= 443:15 +CPU= 38.869
  00:01:09 STEP 1774 H= 443:30 +CPU= 38.871
  00:01:48 STEP 1775 H= 443:45 +CPU= 38.823
  00:02:43 STEP 1776 H= 444:00 +CPU= 54.247
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001776
  00:03:22 STEP 1777 H= 444:15 +CPU= 38.879
  00:04:01 STEP 1778 H= 444:30 +CPU= 38.907
  00:04:40 STEP 1779 H= 444:45 +CPU= 38.942
  00:05:19 STEP 1780 H= 445:00 +CPU= 39.356
  00:05:58 STEP 1781 H= 445:15 +CPU= 38.884
  00:06:37 STEP 1782 H= 445:30 +CPU= 38.839
  00:07:16 STEP 1783 H= 445:45 +CPU= 38.870
  00:07:56 STEP 1784 H= 446:00 +CPU= 39.380
  00:08:35 STEP 1785 H= 446:15 +CPU= 38.826
  00:09:14 STEP 1786 H= 446:30 +CPU= 38.965
  00:09:53 STEP 1787 H= 446:45 +CPU= 38.863
  00:10:42 STEP 1788 H= 447:00 +CPU= 49.078
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001788
  00:11:21 STEP 1789 H= 447:15 +CPU= 38.775
  00:12:00 STEP 1790 H= 447:30 +CPU= 38.768
  00:12:39 STEP 1791 H= 447:45 +CPU= 38.827
  00:13:18 STEP 1792 H= 448:00 +CPU= 39.387
  00:13:57 STEP 1793 H= 448:15 +CPU= 38.868
  00:14:36 STEP 1794 H= 448:30 +CPU= 38.873
  00:15:15 STEP 1795 H= 448:45 +CPU= 38.825
  00:15:54 STEP 1796 H= 449:00 +CPU= 39.259
  00:16:33 STEP 1797 H= 449:15 +CPU= 38.858
  00:17:12 STEP 1798 H= 449:30 +CPU= 38.876
  00:17:51 STEP 1799 H= 449:45 +CPU= 38.760
  00:18:41 STEP 1800 H= 450:00 +CPU= 49.196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001800
  00:19:20 STEP 1801 H= 450:15 +CPU= 38.811
  00:19:59 STEP 1802 H= 450:30 +CPU= 38.958
  00:20:38 STEP 1803 H= 450:45 +CPU= 38.896
  00:21:17 STEP 1804 H= 451:00 +CPU= 39.501
  00:21:56 STEP 1805 H= 451:15 +CPU= 38.767
  00:22:35 STEP 1806 H= 451:30 +CPU= 38.870
  00:23:14 STEP 1807 H= 451:45 +CPU= 38.875
  00:23:54 STEP 1808 H= 452:00 +CPU= 39.455
  00:24:33 STEP 1809 H= 452:15 +CPU= 38.870
  00:25:12 STEP 1810 H= 452:30 +CPU= 38.945
  00:25:51 STEP 1811 H= 452:45 +CPU= 38.933
  00:26:40 STEP 1812 H= 453:00 +CPU= 49.128
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001812
  00:27:19 STEP 1813 H= 453:15 +CPU= 38.827
  00:27:58 STEP 1814 H= 453:30 +CPU= 38.811
  00:28:37 STEP 1815 H= 453:45 +CPU= 38.841
  00:29:17 STEP 1816 H= 454:00 +CPU= 39.427
  00:29:55 STEP 1817 H= 454:15 +CPU= 38.814
  00:30:34 STEP 1818 H= 454:30 +CPU= 38.811
  00:31:13 STEP 1819 H= 454:45 +CPU= 38.826
  00:31:53 STEP 1820 H= 455:00 +CPU= 39.337
  00:32:32 STEP 1821 H= 455:15 +CPU= 38.767
  00:33:11 STEP 1822 H= 455:30 +CPU= 38.767
  00:33:49 STEP 1823 H= 455:45 +CPU= 38.793
  00:34:44 STEP 1824 H= 456:00 +CPU= 54.212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001824
  00:35:23 STEP 1825 H= 456:15 +CPU= 38.904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239_18.zip
Uploading the intermediate file: upload_file_18.zip
  00:36:02 STEP 1826 H= 456:30 +CPU= 38.850
  00:36:41 STEP 1827 H= 456:45 +CPU= 38.887
  00:37:21 STEP 1828 H= 457:00 +CPU= 39.512
  00:37:59 STEP 1829 H= 457:15 +CPU= 38.807
  00:38:39 STEP 1830 H= 457:30 +CPU= 38.951
  00:39:18 STEP 1831 H= 457:45 +CPU= 39.002
  00:39:57 STEP 1832 H= 458:00 +CPU= 39.460
  00:40:36 STEP 1833 H= 458:15 +CPU= 38.922
  00:41:15 STEP 1834 H= 458:30 +CPU= 38.905
  00:41:54 STEP 1835 H= 458:45 +CPU= 38.895
  00:42:43 STEP 1836 H= 459:00 +CPU= 48.792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001836
  00:43:22 STEP 1837 H= 459:15 +CPU= 38.915
  00:44:01 STEP 1838 H= 459:30 +CPU= 38.956
  00:44:40 STEP 1839 H= 459:45 +CPU= 38.976
  00:45:20 STEP 1840 H= 460:00 +CPU= 39.594
  00:45:59 STEP 1841 H= 460:15 +CPU= 38.954
  00:46:38 STEP 1842 H= 460:30 +CPU= 39.059
  00:47:17 STEP 1843 H= 460:45 +CPU= 38.954
  00:47:57 STEP 1844 H= 461:00 +CPU= 39.613
  00:48:36 STEP 1845 H= 461:15 +CPU= 38.991
  00:49:16 STEP 1846 H= 461:30 +CPU= 39.048
  00:49:55 STEP 1847 H= 461:45 +CPU= 38.872
  00:50:44 STEP 1848 H= 462:00 +CPU= 49.137
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001848
  00:51:23 STEP 1849 H= 462:15 +CPU= 38.897
  00:52:02 STEP 1850 H= 462:30 +CPU= 38.869
  00:52:41 STEP 1851 H= 462:45 +CPU= 38.873
  00:53:20 STEP 1852 H= 463:00 +CPU= 39.499
  00:53:59 STEP 1853 H= 463:15 +CPU= 38.915
  00:54:39 STEP 1854 H= 463:30 +CPU= 38.938
  00:55:18 STEP 1855 H= 463:45 +CPU= 38.939
  00:55:57 STEP 1856 H= 464:00 +CPU= 39.468
  00:56:36 STEP 1857 H= 464:15 +CPU= 38.841
  00:57:15 STEP 1858 H= 464:30 +CPU= 38.810
  00:57:54 STEP 1859 H= 464:45 +CPU= 38.857
  00:58:43 STEP 1860 H= 465:00 +CPU= 48.897
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001860
  00:59:22 STEP 1861 H= 465:15 +CPU= 38.627
  01:00:01 STEP 1862 H= 465:30 +CPU= 38.708
  01:00:39 STEP 1863 H= 465:45 +CPU= 38.639
  01:01:19 STEP 1864 H= 466:00 +CPU= 39.285
  01:01:58 STEP 1865 H= 466:15 +CPU= 38.660
  01:02:36 STEP 1866 H= 466:30 +CPU= 38.666
  01:03:15 STEP 1867 H= 466:45 +CPU= 38.717
  01:03:55 STEP 1868 H= 467:00 +CPU= 39.215
  01:04:33 STEP 1869 H= 467:15 +CPU= 38.721
  01:05:12 STEP 1870 H= 467:30 +CPU= 38.731
  01:05:51 STEP 1871 H= 467:45 +CPU= 38.715
  01:06:45 STEP 1872 H= 468:00 +CPU= 54.036
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001872
  01:07:24 STEP 1873 H= 468:15 +CPU= 38.741
  01:08:03 STEP 1874 H= 468:30 +CPU= 38.710
  01:08:42 STEP 1875 H= 468:45 +CPU= 38.730
  01:09:21 STEP 1876 H= 469:00 +CPU= 39.394
  01:10:00 STEP 1877 H= 469:15 +CPU= 38.737
  01:10:39 STEP 1878 H= 469:30 +CPU= 38.825
  01:11:18 STEP 1879 H= 469:45 +CPU= 38.671
  01:11:57 STEP 1880 H= 470:00 +CPU= 39.318
  01:12:36 STEP 1881 H= 470:15 +CPU= 38.786
  01:13:15 STEP 1882 H= 470:30 +CPU= 38.803
  01:13:54 STEP 1883 H= 470:45 +CPU= 38.738
  01:14:43 STEP 1884 H= 471:00 +CPU= 49.032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001884
  01:15:22 STEP 1885 H= 471:15 +CPU= 38.768
  01:16:01 STEP 1886 H= 471:30 +CPU= 38.654
  01:16:40 STEP 1887 H= 471:45 +CPU= 38.798
  01:17:19 STEP 1888 H= 472:00 +CPU= 39.318
  01:17:58 STEP 1889 H= 472:15 +CPU= 38.768
  01:18:37 STEP 1890 H= 472:30 +CPU= 38.701
  01:19:16 STEP 1891 H= 472:45 +CPU= 38.828
  01:19:56 STEP 1892 H= 473:00 +CPU= 39.405
  01:20:34 STEP 1893 H= 473:15 +CPU= 38.707
  01:21:13 STEP 1894 H= 473:30 +CPU= 38.749
  01:21:52 STEP 1895 H= 473:45 +CPU= 38.771
  01:22:42 STEP 1896 H= 474:00 +CPU= 49.198
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001896
  01:23:20 STEP 1897 H= 474:15 +CPU= 38.759
  01:23:59 STEP 1898 H= 474:30 +CPU= 38.743
  01:24:38 STEP 1899 H= 474:45 +CPU= 38.646
  01:25:18 STEP 1900 H= 475:00 +CPU= 39.353
  01:25:56 STEP 1901 H= 475:15 +CPU= 38.747
  01:26:35 STEP 1902 H= 475:30 +CPU= 38.727
  01:27:14 STEP 1903 H= 475:45 +CPU= 38.775
  01:27:54 STEP 1904 H= 476:00 +CPU= 39.403
  01:28:32 STEP 1905 H= 476:15 +CPU= 38.711
  01:29:11 STEP 1906 H= 476:30 +CPU= 38.790
  01:29:50 STEP 1907 H= 476:45 +CPU= 38.682
  01:30:39 STEP 1908 H= 477:00 +CPU= 49.088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001908
  01:31:18 STEP 1909 H= 477:15 +CPU= 38.879
  01:31:57 STEP 1910 H= 477:30 +CPU= 38.838
  01:32:36 STEP 1911 H= 477:45 +CPU= 38.727
  01:33:16 STEP 1912 H= 478:00 +CPU= 39.342
  01:33:54 STEP 1913 H= 478:15 +CPU= 38.643
  01:34:33 STEP 1914 H= 478:30 +CPU= 38.736
  01:35:12 STEP 1915 H= 478:45 +CPU= 38.691
  01:35:52 STEP 1916 H= 479:00 +CPU= 39.374
  01:36:30 STEP 1917 H= 479:15 +CPU= 38.655
  01:37:09 STEP 1918 H= 479:30 +CPU= 38.726
  01:37:48 STEP 1919 H= 479:45 +CPU= 38.699
  01:38:38 STEP 1920 H= 480:00 +CPU= 49.593
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1022968980E-20      I       0.1846048134E-07      I
ICloud fractionI      0.0000000000E+00      I       0.2705730021E-03      I
I  Relat. Hum. I      0.4729989990E+00      I       0.4821764656E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.6204076622E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    325091460 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS650182.9 Mbytes/s, TOTAL TIME=    0.0159( 3.1%)
 01:38:38 STEP 1920 H= 480:00 +CPU= 49.593
 END CNT3
 NSTEP =  1920 CNT0   000000000  35.291  35.291  35.395   0.997               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        11440/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    203    0       46      1044068    11440 ************    32.4     3.8    28.7     4.8    56.3
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.96      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  417    6616.1    6616.1     17.92      0.00
   9 SPCM     - SPECTRAL COMP.                     384    2042.1    2042.1      5.09      0.00
  10 SCAN2M - PHYSICS                              385   17519.8   17519.8     43.80      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 32    1245.0    1245.0      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 17      90.8      90.8      0.01      0.00
  17 GRIDFPOS IN CNT4                               32      39.1      39.1      0.01      0.00
  27 RESTART FILES - WRITING                         8    4980.0    4980.0      0.26      0.00
  28 RESTART FILES - READING                         1    6056.2    6056.2      0.04      0.00
  29 SU4FPOS IN CNT4                                32       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                32    8741.0    8741.0      1.82      0.00
  31 POSDDH IN STEPO                                 2       0.8       0.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    385    8475.2    8475.2     21.19      0.00
  39 SU0YOMB                                         1     316.1     316.1      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    385      22.5      22.5      0.06      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              384     378.4     378.4      0.94      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              384      62.4      62.4      0.16      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 17       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      836     617.1     617.1      3.35      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       800     351.7     351.7      1.83      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        800     147.1     147.1      0.76      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       836     240.3     240.3      1.30      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      50.5      50.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            836      61.3      61.3      0.33      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            800      35.6      35.6      0.19      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            836     106.5     106.5      0.58      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            800      63.5      63.5      0.33      0.00
 400 GSTATS                                      45508       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 42288       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    15398.8 CPU TIME   15351.9 VECTOR TIME    15351.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      112.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289006/ICMGGh7zg+001536
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0_r1628342239_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
01:42:09 (3542): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
17 Jun 2024 08:42:43 1539347 22442581 oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0 1,727,100 78,043 0.0452
16 Jun 2024 15:12:10 1539347 22442581 oifs_43r3_bl_a0iv_2016092300_20_1018_12289006_0 864,000 39,982 0.0463


©2024 climateprediction.net