<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ing IC ancils from: ../../projects/climateprediction.net/jf_abfc551f0103e1ee2f06204f73b75558 to: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d4168fd9dbe57c90c21563de6ebdb85c to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12182193.xml
-- Model is restarting --
Adjusting last_iter, 120, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
last_cpu_time: 4806
upload_file_number: 4
last_iter: 119
last_upload: 345600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338
The child process has been launched with process id: 30313
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30313:30313] [20221222:163717:1671723437:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
16:39:16 STEP 120 H= 120:00 +CPU= 45.759
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000120
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000108
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000108
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_4.zip
Uploading the intermediate file: upload_file_4.zip
16:39:45 STEP 121 H= 121:00 +CPU= 24.443
16:41:31 STEP 122 H= 122:00 +CPU= 31.107
16:42:47 STEP 123 H= 123:00 +CPU= 40.631
16:43:31 STEP 124 H= 124:00 +CPU= 30.135
16:45:58 STEP 125 H= 125:00 +CPU= 33.977
16:48:11 STEP 126 H= 126:00 +CPU= 40.862
16:48:54 STEP 127 H= 127:00 +CPU= 28.552
16:51:11 STEP 128 H= 128:00 +CPU= 34.954
16:55:26 STEP 129 H= 129:00 +CPU= 50.121
16:58:00 STEP 130 H= 130:00 +CPU= 34.503
16:59:04 STEP 131 H= 131:00 +CPU= 30.519
17:03:06 STEP 132 H= 132:00 +CPU= 57.209
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000132
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000132
17:05:27 STEP 133 H= 133:00 +CPU= 34.831
17:06:59 STEP 134 H= 134:00 +CPU= 34.304
17:08:34 STEP 135 H= 135:00 +CPU= 50.460
17:09:17 STEP 136 H= 136:00 +CPU= 25.402
17:10:19 STEP 137 H= 137:00 +CPU= 27.940
17:12:03 STEP 138 H= 138:00 +CPU= 49.263
17:13:54 STEP 139 H= 139:00 +CPU= 32.563
17:15:21 STEP 140 H= 140:00 +CPU= 32.273
17:17:02 STEP 141 H= 141:00 +CPU= 50.666
17:19:05 STEP 142 H= 142:00 +CPU= 36.181
17:20:03 STEP 143 H= 143:00 +CPU= 33.282
17:23:04 STEP 144 H= 144:00 +CPU= 55.100
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000144
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000120
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000132
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000132
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_5.zip
Uploading the intermediate file: upload_file_5.zip
17:25:18 STEP 145 H= 145:00 +CPU= 32.231
17:26:13 STEP 146 H= 146:00 +CPU= 31.472
17:28:15 STEP 147 H= 147:00 +CPU= 51.964
17:29:53 STEP 148 H= 148:00 +CPU= 33.219
17:32:09 STEP 149 H= 149:00 +CPU= 33.716
17:35:17 STEP 150 H= 150:00 +CPU= 51.815
17:36:49 STEP 151 H= 151:00 +CPU= 32.671
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0549
(argv4) batchid: 965
(argv5) wuid: 12182193
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e93aa32ad3adaf4954aecaa1728d8546 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
ic_ancil_file: ic_ancil_12182193
ifsdata_file: ifsdata_12182193
climate_data_file: clim_data_12182193
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_abfc551f0103e1ee2f06204f73b75558 to: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d4168fd9dbe57c90c21563de6ebdb85c to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12182193.xml
-- Model is restarting --
Adjusting last_iter, 152, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
last_cpu_time: 6027
upload_file_number: 6
last_iter: 143
last_upload: 518400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338
The child process has been launched with process id: 30851
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:30851:30851] [20221222:173821:1671727101:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
17:39:41 STEP 144 H= 144:00 +CPU= 46.543
17:40:30 STEP 145 H= 145:00 +CPU= 26.282
17:41:04 STEP 146 H= 146:00 +CPU= 25.167
17:42:49 STEP 147 H= 147:00 +CPU= 47.668
17:44:06 STEP 148 H= 148:00 +CPU= 33.935
17:45:43 STEP 149 H= 149:00 +CPU= 28.590
17:47:10 STEP 150 H= 150:00 +CPU= 47.116
17:48:08 STEP 151 H= 151:00 +CPU= 30.391
17:48:58 STEP 152 H= 152:00 +CPU= 32.833
17:50:35 STEP 153 H= 153:00 +CPU= 50.424
17:51:42 STEP 154 H= 154:00 +CPU= 34.694
17:53:03 STEP 155 H= 155:00 +CPU= 31.678
17:55:25 STEP 156 H= 156:00 +CPU= 54.234
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000156
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000156
17:58:22 STEP 157 H= 157:00 +CPU= 34.201
18:00:05 STEP 158 H= 158:00 +CPU= 33.295
18:02:23 STEP 159 H= 159:00 +CPU= 47.898
18:03:51 STEP 160 H= 160:00 +CPU= 35.270
18:05:36 STEP 161 H= 161:00 +CPU= 33.265
18:06:46 STEP 162 H= 162:00 +CPU= 48.142
18:07:29 STEP 163 H= 163:00 +CPU= 29.002
18:08:30 STEP 164 H= 164:00 +CPU= 33.394
18:10:55 STEP 165 H= 165:00 +CPU= 51.689
18:12:19 STEP 166 H= 166:00 +CPU= 32.368
18:13:25 STEP 167 H= 167:00 +CPU= 31.945
18:15:33 STEP 168 H= 168:00 +CPU= 55.709
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000168
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000144
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000156
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000156
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_6.zip
Uploading the intermediate file: upload_file_6.zip
18:16:57 STEP 169 H= 169:00 +CPU= 32.400
18:18:51 STEP 170 H= 170:00 +CPU= 36.149
18:21:54 STEP 171 H= 171:00 +CPU= 47.960
18:23:31 STEP 172 H= 172:00 +CPU= 32.323
18:25:06 STEP 173 H= 173:00 +CPU= 33.906
18:26:58 STEP 174 H= 174:00 +CPU= 47.199
18:27:52 STEP 175 H= 175:00 +CPU= 33.801
18:28:48 STEP 176 H= 176:00 +CPU= 31.230
18:30:13 STEP 177 H= 177:00 +CPU= 47.582
18:34:06 STEP 178 H= 178:00 +CPU= 35.113
18:35:17 STEP 179 H= 179:00 +CPU= 29.365
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0549
(argv4) batchid: 965
(argv5) wuid: 12182193
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e93aa32ad3adaf4954aecaa1728d8546 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
ic_ancil_file: ic_ancil_12182193
ifsdata_file: ifsdata_12182193
climate_data_file: clim_data_12182193
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_abfc551f0103e1ee2f06204f73b75558 to: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d4168fd9dbe57c90c21563de6ebdb85c to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12182193.xml
-- Model is restarting --
Adjusting last_iter, 180, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
last_cpu_time: 7444
upload_file_number: 7
last_iter: 167
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338
The child process has been launched with process id: 31447
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31447:31447] [20221222:184051:1671730851:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
18:42:06 STEP 168 H= 168:00 +CPU= 42.270
18:42:33 STEP 169 H= 169:00 +CPU= 22.443
18:43:24 STEP 170 H= 170:00 +CPU= 25.593
18:44:20 STEP 171 H= 171:00 +CPU= 40.880
18:46:02 STEP 172 H= 172:00 +CPU= 31.845
18:46:50 STEP 173 H= 173:00 +CPU= 31.170
18:47:41 STEP 174 H= 174:00 +CPU= 44.919
18:48:14 STEP 175 H= 175:00 +CPU= 32.156
18:49:39 STEP 176 H= 176:00 +CPU= 36.054
18:51:07 STEP 177 H= 177:00 +CPU= 47.882
18:52:09 STEP 178 H= 178:00 +CPU= 32.893
18:53:24 STEP 179 H= 179:00 +CPU= 32.867
18:54:22 STEP 180 H= 180:00 +CPU= 56.291
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000180
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000180
18:55:19 STEP 181 H= 181:00 +CPU= 32.691
18:57:13 STEP 182 H= 182:00 +CPU= 35.718
19:00:42 STEP 183 H= 183:00 +CPU= 46.311
19:02:42 STEP 184 H= 184:00 +CPU= 31.528
19:03:27 STEP 185 H= 185:00 +CPU= 31.891
19:04:49 STEP 186 H= 186:00 +CPU= 53.683
19:05:50 STEP 187 H= 187:00 +CPU= 31.058
19:06:53 STEP 188 H= 188:00 +CPU= 32.638
19:10:31 STEP 189 H= 189:00 +CPU= 55.345
19:11:52 STEP 190 H= 190:00 +CPU= 32.928
19:15:01 STEP 191 H= 191:00 +CPU= 35.951
19:17:42 STEP 192 H= 192:00 +CPU= 54.794
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000192
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000180
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_7.zip
Uploading the intermediate file: upload_file_7.zip
19:20:03 STEP 193 H= 193:00 +CPU= 32.389
19:21:37 STEP 194 H= 194:00 +CPU= 33.823
19:23:45 STEP 195 H= 195:00 +CPU= 46.669
19:24:53 STEP 196 H= 196:00 +CPU= 31.329
19:25:45 STEP 197 H= 197:00 +CPU= 33.300
19:26:51 STEP 198 H= 198:00 +CPU= 48.910
19:28:51 STEP 199 H= 199:00 +CPU= 33.261
19:31:13 STEP 200 H= 200:00 +CPU= 34.799
19:33:25 STEP 201 H= 201:00 +CPU= 47.703
19:35:12 STEP 202 H= 202:00 +CPU= 33.182
19:37:40 STEP 203 H= 203:00 +CPU= 32.116
19:39:36 STEP 204 H= 204:00 +CPU= 52.991
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000204
19:40:08 STEP 205 H= 205:00 +CPU= 31.835
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0549
(argv4) batchid: 965
(argv5) wuid: 12182193
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e93aa32ad3adaf4954aecaa1728d8546 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
ic_ancil_file: ic_ancil_12182193
ifsdata_file: ifsdata_12182193
climate_data_file: clim_data_12182193
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_abfc551f0103e1ee2f06204f73b75558 to: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d4168fd9dbe57c90c21563de6ebdb85c to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12182193.xml
-- Model is restarting --
Adjusting last_iter, 206, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
last_cpu_time: 8894
upload_file_number: 8
last_iter: 191
last_upload: 691200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338
The child process has been launched with process id: 31979
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:31979:31979] [20221222:194325:1671734605:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
19:44:33 STEP 192 H= 192:00 +CPU= 49.475
19:45:13 STEP 193 H= 193:00 +CPU= 27.876
19:45:50 STEP 194 H= 194:00 +CPU= 28.191
19:46:37 STEP 195 H= 195:00 +CPU= 44.906
19:47:36 STEP 196 H= 196:00 +CPU= 33.801
19:48:49 STEP 197 H= 197:00 +CPU= 33.152
19:49:55 STEP 198 H= 198:00 +CPU= 44.066
19:50:23 STEP 199 H= 199:00 +CPU= 27.419
19:51:14 STEP 200 H= 200:00 +CPU= 35.400
19:52:12 STEP 201 H= 201:00 +CPU= 50.774
19:53:22 STEP 202 H= 202:00 +CPU= 27.663
19:53:54 STEP 203 H= 203:00 +CPU= 32.278
19:55:00 STEP 204 H= 204:00 +CPU= 57.809
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000204
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000204
19:56:28 STEP 205 H= 205:00 +CPU= 34.471
19:57:03 STEP 206 H= 206:00 +CPU= 33.914
19:58:38 STEP 207 H= 207:00 +CPU= 50.931
19:59:23 STEP 208 H= 208:00 +CPU= 32.033
20:00:30 STEP 209 H= 209:00 +CPU= 32.671
20:01:36 STEP 210 H= 210:00 +CPU= 53.091
20:02:30 STEP 211 H= 211:00 +CPU= 31.699
20:03:27 STEP 212 H= 212:00 +CPU= 28.325
20:05:06 STEP 213 H= 213:00 +CPU= 44.965
20:06:14 STEP 214 H= 214:00 +CPU= 32.396
20:07:02 STEP 215 H= 215:00 +CPU= 31.941
20:09:18 STEP 216 H= 216:00 +CPU= 61.040
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000216
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000204
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_8.zip
Uploading the intermediate file: upload_file_8.zip
20:09:56 STEP 217 H= 217:00 +CPU= 33.257
20:10:45 STEP 218 H= 218:00 +CPU= 30.387
20:11:55 STEP 219 H= 219:00 +CPU= 51.705
20:12:43 STEP 220 H= 220:00 +CPU= 34.665
20:14:41 STEP 221 H= 221:00 +CPU= 33.978
20:16:26 STEP 222 H= 222:00 +CPU= 49.379
20:17:52 STEP 223 H= 223:00 +CPU= 33.298
20:18:55 STEP 224 H= 224:00 +CPU= 34.667
20:20:01 STEP 225 H= 225:00 +CPU= 52.702
20:20:44 STEP 226 H= 226:00 +CPU= 33.377
20:21:39 STEP 227 H= 227:00 +CPU= 33.096
20:22:43 STEP 228 H= 228:00 +CPU= 57.109
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000228
20:23:58 STEP 229 H= 229:00 +CPU= 34.707
20:24:36 STEP 230 H= 230:00 +CPU= 34.325
20:26:24 STEP 231 H= 231:00 +CPU= 50.418
20:27:17 STEP 232 H= 232:00 +CPU= 35.138
20:27:52 STEP 233 H= 233:00 +CPU= 32.228
20:29:22 STEP 234 H= 234:00 +CPU= 54.311
20:30:15 STEP 235 H= 235:00 +CPU= 33.980
20:30:51 STEP 236 H= 236:00 +CPU= 34.285
20:31:51 STEP 237 H= 237:00 +CPU= 48.586
20:32:48 STEP 238 H= 238:00 +CPU= 36.141
20:33:53 STEP 239 H= 239:00 +CPU= 35.321
20:35:05 STEP 240 H= 240:00 +CPU= 56.041
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000228
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_9.zip
Uploading the intermediate file: upload_file_9.zip
20:35:55 STEP 241 H= 241:00 +CPU= 33.415
20:37:19 STEP 242 H= 242:00 +CPU= 35.225
20:38:16 STEP 243 H= 243:00 +CPU= 51.203
20:39:09 STEP 244 H= 244:00 +CPU= 31.447
20:39:43 STEP 245 H= 245:00 +CPU= 34.213
20:40:39 STEP 246 H= 246:00 +CPU= 52.121
20:41:35 STEP 247 H= 247:00 +CPU= 34.950
20:42:23 STEP 248 H= 248:00 +CPU= 35.159
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0549
(argv4) batchid: 965
(argv5) wuid: 12182193
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e93aa32ad3adaf4954aecaa1728d8546 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0549_1996050100_123_965_12182193.zip
ic_ancil_file: ic_ancil_12182193
ifsdata_file: ifsdata_12182193
climate_data_file: clim_data_12182193
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_abfc551f0103e1ee2f06204f73b75558 to: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12182193.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d4168fd9dbe57c90c21563de6ebdb85c to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12182193.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12182193.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12182193.xml
-- Model is restarting --
Adjusting last_iter, 249, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12182193.xml
last_cpu_time: 11167
upload_file_number: 10
last_iter: 239
last_upload: 864000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338
The child process has been launched with process id: 32362
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Aleramo:1:1:32362:32362] [20221222:204458:1671738298:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:46:14 STEP 240 H= 240:00 +CPU= 57.531
20:47:09 STEP 241 H= 241:00 +CPU= 27.473
20:47:40 STEP 242 H= 242:00 +CPU= 29.690
20:48:42 STEP 243 H= 243:00 +CPU= 48.610
20:50:03 STEP 244 H= 244:00 +CPU= 34.813
20:50:41 STEP 245 H= 245:00 +CPU= 33.595
20:51:58 STEP 246 H= 246:00 +CPU= 47.242
20:52:29 STEP 247 H= 247:00 +CPU= 31.251
20:53:05 STEP 248 H= 248:00 +CPU= 34.345
20:54:41 STEP 249 H= 249:00 +CPU= 51.700
20:55:24 STEP 250 H= 250:00 +CPU= 32.653
20:55:58 STEP 251 H= 251:00 +CPU= 32.185
20:57:34 STEP 252 H= 252:00 +CPU= 54.727
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000252
20:58:38 STEP 253 H= 253:00 +CPU= 32.773
20:59:13 STEP 254 H= 254:00 +CPU= 34.764
21:00:34 STEP 255 H= 255:00 +CPU= 49.280
21:01:09 STEP 256 H= 256:00 +CPU= 32.696
21:01:49 STEP 257 H= 257:00 +CPU= 32.138
21:02:56 STEP 258 H= 258:00 +CPU= 50.710
21:03:46 STEP 259 H= 259:00 +CPU= 31.899
21:04:35 STEP 260 H= 260:00 +CPU= 33.307
21:05:35 STEP 261 H= 261:00 +CPU= 51.452
21:06:34 STEP 262 H= 262:00 +CPU= 34.567
21:07:32 STEP 263 H= 263:00 +CPU= 33.488
21:09:07 STEP 264 H= 264:00 +CPU= 58.053
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMGGhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMSHhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182193/ICMUAhq0f+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0549_1996050100_123_965_12182193_0_r1002226338_10.zip
Uploading the intermediate file: upload_file_10.zip
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
