<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021
The child process has been launched with process id: 411086
Executing the command: /var/lib/boinc-client/slots/7/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:411086:411086] [20221129:185815:1669744695:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:00:56 STEP 288 H= 288:00 +CPU= 37.604
19:01:16 STEP 289 H= 289:00 +CPU= 19.171
19:01:36 STEP 290 H= 290:00 +CPU= 19.863
19:02:08 STEP 291 H= 291:00 +CPU= 32.022
19:02:28 STEP 292 H= 292:00 +CPU= 20.199
19:02:47 STEP 293 H= 293:00 +CPU= 18.716
19:03:15 STEP 294 H= 294:00 +CPU= 27.641
19:03:33 STEP 295 H= 295:00 +CPU= 17.743
19:03:53 STEP 296 H= 296:00 +CPU= 19.982
19:04:24 STEP 297 H= 297:00 +CPU= 29.696
19:04:45 STEP 298 H= 298:00 +CPU= 20.759
19:05:25 STEP 299 H= 299:00 +CPU= 21.251
19:06:48 STEP 300 H= 300:00 +CPU= 35.445
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000300
19:07:10 STEP 301 H= 301:00 +CPU= 20.468
19:07:32 STEP 302 H= 302:00 +CPU= 22.281
19:08:06 STEP 303 H= 303:00 +CPU= 33.788
19:08:30 STEP 304 H= 304:00 +CPU= 23.423
19:08:53 STEP 305 H= 305:00 +CPU= 23.342
19:09:44 STEP 306 H= 306:00 +CPU= 31.494
19:10:09 STEP 307 H= 307:00 +CPU= 21.968
19:10:32 STEP 308 H= 308:00 +CPU= 22.550
19:11:23 STEP 309 H= 309:00 +CPU= 30.579
19:11:43 STEP 310 H= 310:00 +CPU= 17.309
Quit request received from BOINC client, ending the child process
double free or corruption (top)
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0158
(argv4) batchid: 945
(argv5) wuid: 12163247
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e63205cb2dde63fcb5d1a6ad225b3d4b to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
ic_ancil_file: ic_ancil_12163247
ifsdata_file: ifsdata_12163247
climate_data_file: clim_data_12163247
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_da3d9dac65eba5b82a7b28dc9154e173 to: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f9e8243b7ac3be5950198b22286fc3c6 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021
The child process has been launched with process id: 6113
Executing the command: /var/lib/boinc-client/slots/7/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:6113:6113] [20221129:192847:1669746527:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0158
(argv4) batchid: 945
(argv5) wuid: 12163247
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e63205cb2dde63fcb5d1a6ad225b3d4b to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
ic_ancil_file: ic_ancil_12163247
ifsdata_file: ifsdata_12163247
climate_data_file: clim_data_12163247
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_da3d9dac65eba5b82a7b28dc9154e173 to: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f9e8243b7ac3be5950198b22286fc3c6 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021
The child process has been launched with process id: 8167
Executing the command: /var/lib/boinc-client/slots/7/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:8167:8167] [20221129:194950:1669747790:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0158
(argv4) batchid: 945
(argv5) wuid: 12163247
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e63205cb2dde63fcb5d1a6ad225b3d4b to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
ic_ancil_file: ic_ancil_12163247
ifsdata_file: ifsdata_12163247
climate_data_file: clim_data_12163247
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_da3d9dac65eba5b82a7b28dc9154e173 to: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f9e8243b7ac3be5950198b22286fc3c6 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021
The child process has been launched with process id: 17518
Executing the command: /var/lib/boinc-client/slots/7/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:17518:17518] [20221129:213124:1669753884:0.125] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:35:33 STEP 288 H= 288:00 +CPU= 42.636
21:35:57 STEP 289 H= 289:00 +CPU= 24.231
21:36:18 STEP 290 H= 290:00 +CPU= 21.449
21:36:51 STEP 291 H= 291:00 +CPU= 32.569
21:37:13 STEP 292 H= 292:00 +CPU= 21.660
21:37:32 STEP 293 H= 293:00 +CPU= 19.767
21:38:02 STEP 294 H= 294:00 +CPU= 29.277
21:38:20 STEP 295 H= 295:00 +CPU= 17.892
21:38:37 STEP 296 H= 296:00 +CPU= 17.555
21:39:05 STEP 297 H= 297:00 +CPU= 27.595
21:39:24 STEP 298 H= 298:00 +CPU= 19.108
21:39:42 STEP 299 H= 299:00 +CPU= 18.342
21:40:17 STEP 300 H= 300:00 +CPU= 28.462
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000300
21:40:35 STEP 301 H= 301:00 +CPU= 17.592
21:40:54 STEP 302 H= 302:00 +CPU= 18.417
21:41:21 STEP 303 H= 303:00 +CPU= 27.415
21:41:42 STEP 304 H= 304:00 +CPU= 19.022
21:42:00 STEP 305 H= 305:00 +CPU= 17.846
21:42:26 STEP 306 H= 306:00 +CPU= 26.400
21:42:45 STEP 307 H= 307:00 +CPU= 16.636
21:43:02 STEP 308 H= 308:00 +CPU= 16.323
21:43:27 STEP 309 H= 309:00 +CPU= 25.014
21:43:44 STEP 310 H= 310:00 +CPU= 16.854
21:44:06 STEP 311 H= 311:00 +CPU= 16.998
21:49:38 STEP 312 H= 312:00 +CPU= 34.497
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000300
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021_1.zip
21:49:58 STEP 313 H= 313:00 +CPU= 19.083
Uploading the intermediate file: upload_file_1.zip
21:50:16 STEP 314 H= 314:00 +CPU= 17.997
Uploading trickle at timestep: 1123200
21:50:45 STEP 315 H= 315:00 +CPU= 28.372
21:51:04 STEP 316 H= 316:00 +CPU= 18.388
21:51:22 STEP 317 H= 317:00 +CPU= 17.840
21:51:51 STEP 318 H= 318:00 +CPU= 28.206
21:52:08 STEP 319 H= 319:00 +CPU= 17.565
21:52:31 STEP 320 H= 320:00 +CPU= 17.625
21:52:58 STEP 321 H= 321:00 +CPU= 27.513
21:53:16 STEP 322 H= 322:00 +CPU= 17.253
21:53:32 STEP 323 H= 323:00 +CPU= 16.586
21:54:06 STEP 324 H= 324:00 +CPU= 29.513
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000324
21:54:24 STEP 325 H= 325:00 +CPU= 18.161
21:54:43 STEP 326 H= 326:00 +CPU= 18.281
21:55:12 STEP 327 H= 327:00 +CPU= 29.423
21:55:32 STEP 328 H= 328:00 +CPU= 19.986
21:55:54 STEP 329 H= 329:00 +CPU= 19.466
21:57:02 STEP 330 H= 330:00 +CPU= 30.271
21:57:20 STEP 331 H= 331:00 +CPU= 17.025
21:57:39 STEP 332 H= 332:00 +CPU= 18.897
21:58:09 STEP 333 H= 333:00 +CPU= 29.967
21:58:27 STEP 334 H= 334:00 +CPU= 17.255
21:58:55 STEP 335 H= 335:00 +CPU= 15.865
22:04:08 STEP 336 H= 336:00 +CPU= 36.929
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000324
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021_2.zip
22:04:27 STEP 337 H= 337:00 +CPU= 17.898
Uploading the intermediate file: upload_file_2.zip
22:04:46 STEP 338 H= 338:00 +CPU= 19.002
Uploading trickle at timestep: 1209600
22:05:15 STEP 339 H= 339:00 +CPU= 28.110
22:05:33 STEP 340 H= 340:00 +CPU= 17.663
22:05:50 STEP 341 H= 341:00 +CPU= 16.945
22:06:16 STEP 342 H= 342:00 +CPU= 26.299
22:06:33 STEP 343 H= 343:00 +CPU= 16.366
22:06:51 STEP 344 H= 344:00 +CPU= 17.134
22:07:16 STEP 345 H= 345:00 +CPU= 25.317
22:07:32 STEP 346 H= 346:00 +CPU= 15.442
22:07:49 STEP 347 H= 347:00 +CPU= 15.484
22:08:18 STEP 348 H= 348:00 +CPU= 26.114
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000348
22:08:33 STEP 349 H= 349:00 +CPU= 14.016
22:08:48 STEP 350 H= 350:00 +CPU= 14.875
22:09:10 STEP 351 H= 351:00 +CPU= 21.518
22:09:24 STEP 352 H= 352:00 +CPU= 13.384
22:09:38 STEP 353 H= 353:00 +CPU= 13.125
22:09:59 STEP 354 H= 354:00 +CPU= 20.844
22:10:12 STEP 355 H= 355:00 +CPU= 12.917
22:10:26 STEP 356 H= 356:00 +CPU= 13.636
22:10:47 STEP 357 H= 357:00 +CPU= 20.550
22:11:01 STEP 358 H= 358:00 +CPU= 13.667
22:11:55 STEP 359 H= 359:00 +CPU= 13.800
22:16:57 STEP 360 H= 360:00 +CPU= 35.966
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021_3.zip
Uploading the intermediate file: upload_file_3.zip
22:17:22 STEP 361 H= 361:00 +CPU= 24.064
Uploading trickle at timestep: 1296000
22:17:49 STEP 362 H= 362:00 +CPU= 26.927
22:18:29 STEP 363 H= 363:00 +CPU= 39.312
22:19:01 STEP 364 H= 364:00 +CPU= 32.075
22:19:35 STEP 365 H= 365:00 +CPU= 33.279
22:20:25 STEP 366 H= 366:00 +CPU= 49.234
22:20:54 STEP 367 H= 367:00 +CPU= 28.814
22:21:27 STEP 368 H= 368:00 +CPU= 32.447
22:22:17 STEP 369 H= 369:00 +CPU= 49.948
22:22:52 STEP 370 H= 370:00 +CPU= 35.036
22:23:24 STEP 371 H= 371:00 +CPU= 31.991
22:24:19 STEP 372 H= 372:00 +CPU= 54.269
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000372
22:24:52 STEP 373 H= 373:00 +CPU= 33.446
22:25:26 STEP 374 H= 374:00 +CPU= 33.091
22:26:18 STEP 375 H= 375:00 +CPU= 52.346
22:26:49 STEP 376 H= 376:00 +CPU= 31.205
22:27:20 STEP 377 H= 377:00 +CPU= 30.199
22:28:07 STEP 378 H= 378:00 +CPU= 46.267
22:28:38 STEP 379 H= 379:00 +CPU= 30.931
22:29:10 STEP 380 H= 380:00 +CPU= 31.880
22:29:56 STEP 381 H= 381:00 +CPU= 45.911
22:30:32 STEP 382 H= 382:00 +CPU= 35.887
22:31:05 STEP 383 H= 383:00 +CPU= 33.117
22:32:04 STEP 384 H= 384:00 +CPU= 57.983
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMGGhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMSHhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247/ICMUAhpi1+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021_4.zip
Uploading the intermediate file: upload_file_4.zip
Uploading trickle at timestep: 1382400
22:32:34 STEP 385 H= 385:00 +CPU= 30.005
22:33:06 STEP 386 H= 386:00 +CPU= 31.923
22:33:54 STEP 387 H= 387:00 +CPU= 47.689
22:34:25 STEP 388 H= 388:00 +CPU= 30.498
22:34:55 STEP 389 H= 389:00 +CPU= 29.375
22:35:46 STEP 390 H= 390:00 +CPU= 51.300
22:36:19 STEP 391 H= 391:00 +CPU= 32.616
22:36:52 STEP 392 H= 392:00 +CPU= 33.331
22:37:44 STEP 393 H= 393:00 +CPU= 51.482
22:38:14 STEP 394 H= 394:00 +CPU= 30.090
22:38:47 STEP 395 H= 395:00 +CPU= 32.698
22:39:37 STEP 396 H= 396:00 +CPU= 50.149
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhpi1+000396
22:40:09 STEP 397 H= 397:00 +CPU= 31.022
22:40:38 STEP 398 H= 398:00 +CPU= 28.603
22:41:26 STEP 399 H= 399:00 +CPU= 48.267
22:41:57 STEP 400 H= 400:00 +CPU= 30.881
22:42:26 STEP 401 H= 401:00 +CPU= 28.410
22:43:08 STEP 402 H= 402:00 +CPU= 42.648
22:43:37 STEP 403 H= 403:00 +CPU= 28.308
22:44:03 STEP 404 H= 404:00 +CPU= 26.554
22:44:47 STEP 405 H= 405:00 +CPU= 41.925
22:45:17 STEP 406 H= 406:00 +CPU= 29.926
22:46:02 STEP 407 H= 407:00 +CPU= 29.504
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0158
(argv4) batchid: 945
(argv5) wuid: 12163247
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163247
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e63205cb2dde63fcb5d1a6ad225b3d4b to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0158_2021050100_123_945_12163247.zip
ic_ancil_file: ic_ancil_12163247
ifsdata_file: ifsdata_12163247
climate_data_file: clim_data_12163247
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_da3d9dac65eba5b82a7b28dc9154e173 to: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12163247.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f9e8243b7ac3be5950198b22286fc3c6 to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12163247.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12163247.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0158_2021050100_123_945_12163247_0_r698163021
The child process has been launched with process id: 36418
Executing the command: /var/lib/boinc-client/slots/7/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105219:1669801939:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.393] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/7/./master.exe' : sigcontextptr=0x7fff7bc8afa0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/7/./master.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x7f00008e42], nsigs = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105306:1669801986:47.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105307:1669801987:48.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:johann:1:1:36418:36418] [20221130:105307:1669801987:48.405] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
master.exe 0000000001CE916B Unknown Unknown Unknown
master.exe 000000000081FFF1 Unknown Unknown Unknown
master.exe 0000000001DC9090 Unknown Unknown Unknown
master.exe 0000000001DC903B Unknown Unknown Unknown
master.exe 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
master.exe 000000000081D8AF abor1_ 41 abor1.F90
master.exe 00000000015F8ACA mpabort_ 50 mpabort.F
master.exe 00000000015D51C8 abort1_ 44 abort1.F
master.exe 000000000166B07F userin_ 553 userin.F
master.exe 00000000015D7FF0 initmdl_ 452 initmdl.F
master.exe 00000000008711E0 wavemdl_ 393 wavemdl.F
master.exe 00000000007E0165 wvcouple_ 260 wvcouple.F90
master.exe 000000000055E4AD cnt4_ 1283 cnt4.F90
master.exe 00000000005471C9 cnt3_ 267 cnt3.F90
master.exe 0000000000546412 cnt2_ 88 cnt2.F90
master.exe 0000000000545E28 cnt1_ 92 cnt1.F90
master.exe 000000000040708D cnt0_ 146 cnt0.F90
master.exe 000000000040220F MAIN__ 96 master.F90
master.exe 00000000004021A2 Unknown Unknown Unknown
master.exe 0000000001DCA390 Unknown Unknown Unknown
master.exe 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
