<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
hub/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002)
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : POSTPHY_LAYER
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : POSTPHY
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130740:1674151660:6377.921] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CPPHDDHE
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130741:1674151661:6378.923] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3208:3208] [20230119:130741:1674151661:6378.923] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000ECB221 cpphddhe_ 310 cpphddhe.F90
oifs_43r3_model.e 0000000000EC7BB0 postphy_ 1033 postphy.F90
oifs_43r3_model.e 0000000000E9A3FB postphy_layer_ 144 postphy_layer.F90
oifs_43r3_model.e 0000000000E5022E ec_phys_ 945 ec_phys.F90
oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0514
(argv4) batchid: 989
(argv5) wuid: 12206158
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
ic_ancil_file: ic_ancil_12206158
ifsdata_file: ifsdata_12206158
climate_data_file: clim_data_12206158
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml
-- Model is restarting --
Adjusting last_iter, 2607, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
last_cpu_time: 50131
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037
The child process has been launched with process id: 3143
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3143:3143] [20230119:130911:1674151751:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0514
(argv4) batchid: 989
(argv5) wuid: 12206158
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
ic_ancil_file: ic_ancil_12206158
ifsdata_file: ifsdata_12206158
climate_data_file: clim_data_12206158
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml
-- Model is restarting --
Adjusting last_iter, 2592, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
last_cpu_time: 50150
upload_file_number: 108
last_iter: 2591
last_upload: 9331200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037
The child process has been launched with process id: 3068
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:084250:1674222170:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:43:54 STEP 2592 H=2592:00 +CPU= 34.969
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:44:22 STEP 2593 H=2593:00 +CPU= 16.821
08:44:40 STEP 2594 H=2594:00 +CPU= 17.004
08:45:06 STEP 2595 H=2595:00 +CPU= 26.459
08:45:24 STEP 2596 H=2596:00 +CPU= 17.526
08:45:42 STEP 2597 H=2597:00 +CPU= 17.431
08:46:09 STEP 2598 H=2598:00 +CPU= 26.643
08:46:26 STEP 2599 H=2599:00 +CPU= 16.812
08:46:44 STEP 2600 H=2600:00 +CPU= 17.598
08:47:10 STEP 2601 H=2601:00 +CPU= 25.866
08:47:27 STEP 2602 H=2602:00 +CPU= 16.948
08:47:45 STEP 2603 H=2603:00 +CPU= 16.712
08:48:13 STEP 2604 H=2604:00 +CPU= 28.055
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
08:48:30 STEP 2605 H=2605:00 +CPU= 16.742
08:48:47 STEP 2606 H=2606:00 +CPU= 16.944
08:49:13 STEP 2607 H=2607:00 +CPU= 25.544
08:49:30 STEP 2608 H=2608:00 +CPU= 17.259
08:49:49 STEP 2609 H=2609:00 +CPU= 18.606
08:50:19 STEP 2610 H=2610:00 +CPU= 29.405
08:50:37 STEP 2611 H=2611:00 +CPU= 17.110
08:50:56 STEP 2612 H=2612:00 +CPU= 19.415
08:51:25 STEP 2613 H=2613:00 +CPU= 27.575
08:51:41 STEP 2614 H=2614:00 +CPU= 16.803
08:51:59 STEP 2615 H=2615:00 +CPU= 16.930
08:52:30 STEP 2616 H=2616:00 +CPU= 31.041
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_108.zip
Uploading the intermediate file: upload_file_108.zip
08:52:47 STEP 2617 H=2617:00 +CPU= 17.221
08:53:05 STEP 2618 H=2618:00 +CPU= 17.189
08:53:31 STEP 2619 H=2619:00 +CPU= 25.861
08:53:33 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
08:53:47 STEP 2620 H=2620:00 +CPU= 16.225
08:54:05 STEP 2621 H=2621:00 +CPU= 16.969
08:54:31 STEP 2622 H=2622:00 +CPU= 25.688
08:54:47 STEP 2623 H=2623:00 +CPU= 15.690
08:55:04 STEP 2624 H=2624:00 +CPU= 16.764
08:55:29 STEP 2625 H=2625:00 +CPU= 25.542
08:55:47 STEP 2626 H=2626:00 +CPU= 17.417
08:56:03 STEP 2627 H=2627:00 +CPU= 16.103
08:56:32 STEP 2628 H=2628:00 +CPU= 28.467
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
08:56:48 STEP 2629 H=2629:00 +CPU= 15.995
08:57:04 STEP 2630 H=2630:00 +CPU= 15.941
08:57:32 STEP 2631 H=2631:00 +CPU= 26.707
08:57:51 STEP 2632 H=2632:00 +CPU= 18.890
08:58:10 STEP 2633 H=2633:00 +CPU= 18.168
08:58:39 STEP 2634 H=2634:00 +CPU= 28.278
08:58:58 STEP 2635 H=2635:00 +CPU= 18.139
08:59:17 STEP 2636 H=2636:00 +CPU= 18.523
08:59:44 STEP 2637 H=2637:00 +CPU= 27.363
09:00:03 STEP 2638 H=2638:00 +CPU= 18.339
09:00:22 STEP 2639 H=2639:00 +CPU= 18.459
09:00:58 STEP 2640 H=2640:00 +CPU= 32.372
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_109.zip
Uploading the intermediate file: upload_file_109.zip
09:01:17 STEP 2641 H=2641:00 +CPU= 18.251
09:01:35 STEP 2642 H=2642:00 +CPU= 17.976
09:02:00 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:02:03 STEP 2643 H=2643:00 +CPU= 27.404
09:02:22 STEP 2644 H=2644:00 +CPU= 18.474
09:02:40 STEP 2645 H=2645:00 +CPU= 17.619
09:03:08 STEP 2646 H=2646:00 +CPU= 27.989
09:03:27 STEP 2647 H=2647:00 +CPU= 17.772
09:03:45 STEP 2648 H=2648:00 +CPU= 18.164
09:04:14 STEP 2649 H=2649:00 +CPU= 28.068
09:04:33 STEP 2650 H=2650:00 +CPU= 18.354
09:04:52 STEP 2651 H=2651:00 +CPU= 18.622
09:05:23 STEP 2652 H=2652:00 +CPU= 30.751
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652
09:05:42 STEP 2653 H=2653:00 +CPU= 18.391
09:06:02 STEP 2654 H=2654:00 +CPU= 19.220
09:06:31 STEP 2655 H=2655:00 +CPU= 28.035
09:06:50 STEP 2656 H=2656:00 +CPU= 18.989
09:07:09 STEP 2657 H=2657:00 +CPU= 18.522
09:07:38 STEP 2658 H=2658:00 +CPU= 28.171
09:07:57 STEP 2659 H=2659:00 +CPU= 18.434
09:08:17 STEP 2660 H=2660:00 +CPU= 19.789
09:08:46 STEP 2661 H=2661:00 +CPU= 27.749
09:09:06 STEP 2662 H=2662:00 +CPU= 19.250
09:09:25 STEP 2663 H=2663:00 +CPU= 18.808
09:09:59 STEP 2664 H=2664:00 +CPU= 33.000
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_110.zip
Uploading the intermediate file: upload_file_110.zip
09:10:18 STEP 2665 H=2665:00 +CPU= 18.708
09:10:38 STEP 2666 H=2666:00 +CPU= 19.545
09:11:04 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:11:08 STEP 2667 H=2667:00 +CPU= 29.472
09:11:28 STEP 2668 H=2668:00 +CPU= 19.127
09:11:47 STEP 2669 H=2669:00 +CPU= 18.840
09:12:16 STEP 2670 H=2670:00 +CPU= 28.376
09:12:36 STEP 2671 H=2671:00 +CPU= 18.726
09:12:56 STEP 2672 H=2672:00 +CPU= 19.481
09:13:25 STEP 2673 H=2673:00 +CPU= 29.026
09:13:45 STEP 2674 H=2674:00 +CPU= 19.354
09:14:04 STEP 2675 H=2675:00 +CPU= 18.580
09:14:37 STEP 2676 H=2676:00 +CPU= 31.220
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676
09:14:56 STEP 2677 H=2677:00 +CPU= 19.117
09:15:16 STEP 2678 H=2678:00 +CPU= 19.126
09:15:45 STEP 2679 H=2679:00 +CPU= 28.126
09:16:05 STEP 2680 H=2680:00 +CPU= 19.326
09:16:24 STEP 2681 H=2681:00 +CPU= 18.446
09:16:53 STEP 2682 H=2682:00 +CPU= 28.678
09:17:12 STEP 2683 H=2683:00 +CPU= 18.556
09:17:31 STEP 2684 H=2684:00 +CPU= 18.827
09:18:00 STEP 2685 H=2685:00 +CPU= 27.974
09:18:19 STEP 2686 H=2686:00 +CPU= 18.710
09:18:40 STEP 2687 H=2687:00 +CPU= 19.790
09:19:14 STEP 2688 H=2688:00 +CPU= 33.775
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_111.zip
Uploading the intermediate file: upload_file_111.zip
09:19:33 STEP 2689 H=2689:00 +CPU= 18.451
09:19:53 STEP 2690 H=2690:00 +CPU= 19.656
09:20:22 STEP 2691 H=2691:00 +CPU= 28.453
09:20:23 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:20:42 STEP 2692 H=2692:00 +CPU= 18.965
09:21:01 STEP 2693 H=2693:00 +CPU= 18.700
09:21:31 STEP 2694 H=2694:00 +CPU= 28.710
09:21:50 STEP 2695 H=2695:00 +CPU= 18.605
09:22:10 STEP 2696 H=2696:00 +CPU= 19.547
09:22:39 STEP 2697 H=2697:00 +CPU= 28.497
09:22:59 STEP 2698 H=2698:00 +CPU= 19.110
09:23:18 STEP 2699 H=2699:00 +CPU= 18.849
09:23:51 STEP 2700 H=2700:00 +CPU= 31.460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700
09:24:10 STEP 2701 H=2701:00 +CPU= 18.516
09:24:29 STEP 2702 H=2702:00 +CPU= 19.059
09:24:59 STEP 2703 H=2703:00 +CPU= 28.492
09:25:19 STEP 2704 H=2704:00 +CPU= 19.201
09:25:38 STEP 2705 H=2705:00 +CPU= 18.813
09:26:07 STEP 2706 H=2706:00 +CPU= 29.004
09:26:26 STEP 2707 H=2707:00 +CPU= 18.423
09:26:45 STEP 2708 H=2708:00 +CPU= 18.602
09:27:14 STEP 2709 H=2709:00 +CPU= 28.299
09:27:33 STEP 2710 H=2710:00 +CPU= 18.714
09:27:53 STEP 2711 H=2711:00 +CPU= 19.138
09:28:28 STEP 2712 H=2712:00 +CPU= 34.138
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_112.zip
Uploading the intermediate file: upload_file_112.zip
09:28:47 STEP 2713 H=2713:00 +CPU= 18.112
09:29:07 STEP 2714 H=2714:00 +CPU= 19.331
09:29:34 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:29:36 STEP 2715 H=2715:00 +CPU= 28.300
09:29:55 STEP 2716 H=2716:00 +CPU= 18.732
09:30:14 STEP 2717 H=2717:00 +CPU= 18.539
09:30:44 STEP 2718 H=2718:00 +CPU= 29.211
09:31:04 STEP 2719 H=2719:00 +CPU= 18.762
09:31:23 STEP 2720 H=2720:00 +CPU= 19.100
09:31:52 STEP 2721 H=2721:00 +CPU= 27.867
09:32:11 STEP 2722 H=2722:00 +CPU= 19.087
09:32:31 STEP 2723 H=2723:00 +CPU= 19.251
09:33:02 STEP 2724 H=2724:00 +CPU= 30.338
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002724
09:33:21 STEP 2725 H=2725:00 +CPU= 18.451
09:33:41 STEP 2726 H=2726:00 +CPU= 19.103
09:34:10 STEP 2727 H=2727:00 +CPU= 28.035
09:34:30 STEP 2728 H=2728:00 +CPU= 19.183
09:34:49 STEP 2729 H=2729:00 +CPU= 18.469
09:35:17 STEP 2730 H=2730:00 +CPU= 27.841
09:35:36 STEP 2731 H=2731:00 +CPU= 18.792
09:35:56 STEP 2732 H=2732:00 +CPU= 19.309
09:36:25 STEP 2733 H=2733:00 +CPU= 28.209
09:36:44 STEP 2734 H=2734:00 +CPU= 18.719
09:37:03 STEP 2735 H=2735:00 +CPU= 18.575
09:37:38 STEP 2736 H=2736:00 +CPU= 33.431
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_113.zip
Uploading the intermediate file: upload_file_113.zip
09:37:58 STEP 2737 H=2737:00 +CPU= 18.598
09:38:17 STEP 2738 H=2738:00 +CPU= 19.041
09:38:37 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:38:46 STEP 2739 H=2739:00 +CPU= 28.213
09:39:05 STEP 2740 H=2740:00 +CPU= 18.261
09:39:24 STEP 2741 H=2741:00 +CPU= 18.371
09:39:53 STEP 2742 H=2742:00 +CPU= 28.822
09:40:12 STEP 2743 H=2743:00 +CPU= 18.121
09:40:31 STEP 2744 H=2744:00 +CPU= 18.634
09:41:01 STEP 2745 H=2745:00 +CPU= 28.727
09:41:20 STEP 2746 H=2746:00 +CPU= 18.792
09:41:39 STEP 2747 H=2747:00 +CPU= 18.633
09:42:11 STEP 2748 H=2748:00 +CPU= 30.784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002748
09:42:29 STEP 2749 H=2749:00 +CPU= 18.046
09:42:49 STEP 2750 H=2750:00 +CPU= 18.749
09:43:18 STEP 2751 H=2751:00 +CPU= 28.654
09:43:38 STEP 2752 H=2752:00 +CPU= 18.988
09:43:57 STEP 2753 H=2753:00 +CPU= 18.570
09:44:26 STEP 2754 H=2754:00 +CPU= 28.412
09:44:45 STEP 2755 H=2755:00 +CPU= 18.544
09:45:04 STEP 2756 H=2756:00 +CPU= 18.870
09:45:34 STEP 2757 H=2757:00 +CPU= 28.314
09:45:53 STEP 2758 H=2758:00 +CPU= 18.739
09:46:13 STEP 2759 H=2759:00 +CPU= 19.051
09:46:47 STEP 2760 H=2760:00 +CPU= 32.967
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_114.zip
Uploading the intermediate file: upload_file_114.zip
09:47:06 STEP 2761 H=2761:00 +CPU= 18.462
09:47:25 STEP 2762 H=2762:00 +CPU= 18.802
09:47:43 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:47:54 STEP 2763 H=2763:00 +CPU= 28.099
09:48:14 STEP 2764 H=2764:00 +CPU= 19.127
09:48:32 STEP 2765 H=2765:00 +CPU= 18.088
09:49:01 STEP 2766 H=2766:00 +CPU= 28.712
09:49:20 STEP 2767 H=2767:00 +CPU= 17.992
09:49:39 STEP 2768 H=2768:00 +CPU= 18.513
09:50:08 STEP 2769 H=2769:00 +CPU= 27.977
09:50:27 STEP 2770 H=2770:00 +CPU= 18.684
09:50:45 STEP 2771 H=2771:00 +CPU= 17.961
09:51:17 STEP 2772 H=2772:00 +CPU= 30.874
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002772
09:51:35 STEP 2773 H=2773:00 +CPU= 18.022
09:51:55 STEP 2774 H=2774:00 +CPU= 18.628
09:52:24 STEP 2775 H=2775:00 +CPU= 28.273
09:52:43 STEP 2776 H=2776:00 +CPU= 18.979
09:53:02 STEP 2777 H=2777:00 +CPU= 18.305
09:53:31 STEP 2778 H=2778:00 +CPU= 28.758
09:53:50 STEP 2779 H=2779:00 +CPU= 18.218
09:54:10 STEP 2780 H=2780:00 +CPU= 19.176
09:54:39 STEP 2781 H=2781:00 +CPU= 28.751
09:54:59 STEP 2782 H=2782:00 +CPU= 18.678
09:55:18 STEP 2783 H=2783:00 +CPU= 18.923
09:55:53 STEP 2784 H=2784:00 +CPU= 33.717
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_115.zip
Uploading the intermediate file: upload_file_115.zip
09:56:11 STEP 2785 H=2785:00 +CPU= 18.216
09:56:31 STEP 2786 H=2786:00 +CPU= 19.052
09:56:58 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
09:56:59 STEP 2787 H=2787:00 +CPU= 27.732
09:57:19 STEP 2788 H=2788:00 +CPU= 18.703
09:57:37 STEP 2789 H=2789:00 +CPU= 18.324
09:58:07 STEP 2790 H=2790:00 +CPU= 28.660
09:58:25 STEP 2791 H=2791:00 +CPU= 17.952
09:58:44 STEP 2792 H=2792:00 +CPU= 18.653
09:59:13 STEP 2793 H=2793:00 +CPU= 28.170
09:59:32 STEP 2794 H=2794:00 +CPU= 18.762
09:59:51 STEP 2795 H=2795:00 +CPU= 18.425
10:00:23 STEP 2796 H=2796:00 +CPU= 31.083
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002796
10:00:42 STEP 2797 H=2797:00 +CPU= 18.073
10:01:01 STEP 2798 H=2798:00 +CPU= 18.816
10:01:26 STEP 2799 H=2799:00 +CPU= 24.566
10:01:42 STEP 2800 H=2800:00 +CPU= 15.406
10:01:57 STEP 2801 H=2801:00 +CPU= 15.537
10:02:22 STEP 2802 H=2802:00 +CPU= 24.778
10:02:37 STEP 2803 H=2803:00 +CPU= 15.173
10:02:53 STEP 2804 H=2804:00 +CPU= 15.268
10:03:17 STEP 2805 H=2805:00 +CPU= 24.025
10:03:32 STEP 2806 H=2806:00 +CPU= 15.377
10:03:48 STEP 2807 H=2807:00 +CPU= 16.108
10:04:16 STEP 2808 H=2808:00 +CPU= 27.753
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_116.zip
Uploading the intermediate file: upload_file_116.zip
10:04:31 STEP 2809 H=2809:00 +CPU= 14.917
10:04:47 STEP 2810 H=2810:00 +CPU= 16.102
10:05:12 STEP 2811 H=2811:00 +CPU= 24.127
10:05:15 (2306): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
10:05:27 STEP 2812 H=2812:00 +CPU= 15.249
10:05:42 STEP 2813 H=2813:00 +CPU= 15.438
10:06:07 STEP 2814 H=2814:00 +CPU= 24.163
10:06:21 STEP 2815 H=2815:00 +CPU= 14.831
10:06:37 STEP 2816 H=2816:00 +CPU= 15.721
10:07:01 STEP 2817 H=2817:00 +CPU= 23.193
10:07:16 STEP 2818 H=2818:00 +CPU= 15.363
10:07:31 STEP 2819 H=2819:00 +CPU= 15.015
10:07:57 STEP 2820 H=2820:00 +CPU= 25.967
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820
10:08:13 STEP 2821 H=2821:00 +CPU= 15.440
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.660] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2550MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2598 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAMODEL
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-WAMODEL 2 (1431)
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IMPLSCH
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100828:1674227308:5137.661] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SNONLIN
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100829:1674227309:5138.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:rainbow-ubnt:1:1:3068:3068] [20230120:100829:1674227309:5138.661] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001A6C650 snonlin_ 271 snonlin.F
oifs_43r3_model.e 0000000001A31273 implsch_ 285 implsch.F
oifs_43r3_model.e 000000000168CBD7 wamodel_ 1008 wamodel.F
oifs_43r3_model.e 0000000000872295 wavemdl_ 584 wavemdl.F
oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90
oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0514
(argv4) batchid: 989
(argv5) wuid: 12206158
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_950ee5a026717edb7679c775bdd36e2f to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0514_2020050100_123_989_12206158.zip
ic_ancil_file: ic_ancil_12206158
ifsdata_file: ifsdata_12206158
climate_data_file: clim_data_12206158
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4ead1188f398400684c6ffb3e68f2703 to: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12206158.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_63a9cb1954bf64a591cb5b44413d64f7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12206158.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12206158.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12206158.xml
-- Model is restarting --
Adjusting last_iter, 2822, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12206158.xml
last_cpu_time: 55130
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037
The child process has been launched with process id: 2839
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:rainbow-ubnt:1:1:2839:2839] [20230120:123559:1674236159:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:36:52 STEP 2808 H=2808:00 +CPU= 32.904
12:37:09 STEP 2809 H=2809:00 +CPU= 16.540
12:37:25 STEP 2810 H=2810:00 +CPU= 16.582
12:37:50 STEP 2811 H=2811:00 +CPU= 24.274
12:38:06 STEP 2812 H=2812:00 +CPU= 15.990
12:38:22 STEP 2813 H=2813:00 +CPU= 16.071
12:38:46 STEP 2814 H=2814:00 +CPU= 24.490
12:39:02 STEP 2815 H=2815:00 +CPU= 15.442
12:39:18 STEP 2816 H=2816:00 +CPU= 15.609
12:39:42 STEP 2817 H=2817:00 +CPU= 24.190
12:39:57 STEP 2818 H=2818:00 +CPU= 15.675
12:40:13 STEP 2819 H=2819:00 +CPU= 15.986
12:40:40 STEP 2820 H=2820:00 +CPU= 26.748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820
12:40:56 STEP 2821 H=2821:00 +CPU= 15.634
12:41:12 STEP 2822 H=2822:00 +CPU= 15.739
12:41:37 STEP 2823 H=2823:00 +CPU= 24.846
12:41:52 STEP 2824 H=2824:00 +CPU= 15.493
12:42:08 STEP 2825 H=2825:00 +CPU= 15.680
12:42:33 STEP 2826 H=2826:00 +CPU= 24.995
12:42:49 STEP 2827 H=2827:00 +CPU= 15.644
12:43:05 STEP 2828 H=2828:00 +CPU= 15.938
12:43:34 STEP 2829 H=2829:00 +CPU= 28.154
12:43:53 STEP 2830 H=2830:00 +CPU= 19.033
12:44:11 STEP 2831 H=2831:00 +CPU= 17.680
12:44:44 STEP 2832 H=2832:00 +CPU= 31.893
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206158/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0514_2020050100_123_989_12206158_0_r1623507037_117.zip
Uploading the intermediate file: upload_file_117.zip
12:45:03 STEP 2833 H=2833:00 +CPU= 17.226
12:45:20 STEP 2834 H=2834:00 +CPU= 16.909
12:45:45 STEP 2835 H=2835:00 +CPU= 25.456
12:45:50 (2323): handle_upload_file_status: can't parse boinc_ufs_upload_file_93.zip
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
