climateprediction.net home page
Task 22246932

Task 22246932

Name oifs_43r3_ps_0379_2021050100_123_945_12163468_1
Workunit 12163468
Created 29 Nov 2022, 12:26:19 UTC
Sent 30 Nov 2022, 11:53:48 UTC
Report deadline 30 Dec 2022, 11:53:48 UTC
Received 9 Dec 2022, 8:22:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536378
Run time 11 hours 19 min 47 sec
CPU time 55 min 20 sec
Validate state Invalid
Credit 2,056.32
Device peak FLOPS 3.04 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,409.99 MB
Peak swap size 4,978.07 MB
Peak disk usage 9,345.16 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ocess_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267193:267193] [20221209:025805:1670551085:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:00:13 STEP  240 H= 240:00 +CPU=109.116
  03:01:16 STEP  241 H= 241:00 +CPU= 61.818
  03:02:19 STEP  242 H= 242:00 +CPU= 62.585
  03:03:58 STEP  243 H= 243:00 +CPU= 98.744
  03:05:02 STEP  244 H= 244:00 +CPU= 63.879
  03:06:03 STEP  245 H= 245:00 +CPU= 60.700
  03:07:41 STEP  246 H= 246:00 +CPU= 98.512
  03:08:44 STEP  247 H= 247:00 +CPU= 62.021
  03:09:46 STEP  248 H= 248:00 +CPU= 61.638
  03:11:24 STEP  249 H= 249:00 +CPU= 97.446
  03:12:28 STEP  250 H= 250:00 +CPU= 64.413
  03:13:29 STEP  251 H= 251:00 +CPU= 61.147
  03:15:20 STEP  252 H= 252:00 +CPU=109.883
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000252
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000252
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000252
  03:16:22 STEP  253 H= 253:00 +CPU= 61.502
  03:17:27 STEP  254 H= 254:00 +CPU= 63.820
  03:19:04 STEP  255 H= 255:00 +CPU= 95.740
  03:20:07 STEP  256 H= 256:00 +CPU= 62.319
  03:21:08 STEP  257 H= 257:00 +CPU= 61.175
  03:22:48 STEP  258 H= 258:00 +CPU= 99.543
  03:23:50 STEP  259 H= 259:00 +CPU= 61.578
  03:24:54 STEP  260 H= 260:00 +CPU= 63.781
  03:26:31 STEP  261 H= 261:00 +CPU= 96.905
  03:27:34 STEP  262 H= 262:00 +CPU= 62.435
  03:28:39 STEP  263 H= 263:00 +CPU= 63.753
  03:30:33 STEP  264 H= 264:00 +CPU=112.389
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000264
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000264
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 950400
  03:31:36 STEP  265 H= 265:00 +CPU= 62.033
  03:32:37 STEP  266 H= 266:00 +CPU= 60.851
  03:34:08 STEP  267 H= 267:00 +CPU= 91.166
  03:35:09 STEP  268 H= 268:00 +CPU= 61.209
  03:36:09 STEP  269 H= 269:00 +CPU= 59.887
  03:37:48 STEP  270 H= 270:00 +CPU= 97.330
  03:38:48 STEP  271 H= 271:00 +CPU= 59.966
  03:39:50 STEP  272 H= 272:00 +CPU= 61.604
  03:41:24 STEP  273 H= 273:00 +CPU= 93.612
  03:42:26 STEP  274 H= 274:00 +CPU= 61.532
  03:43:26 STEP  275 H= 275:00 +CPU= 60.103
  03:45:17 STEP  276 H= 276:00 +CPU=108.602
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000276
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000276
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000276
  03:46:17 STEP  277 H= 277:00 +CPU= 59.978
  03:47:18 STEP  278 H= 278:00 +CPU= 61.189
  03:48:53 STEP  279 H= 279:00 +CPU= 94.571
  03:49:55 STEP  280 H= 280:00 +CPU= 61.355
  03:50:54 STEP  281 H= 281:00 +CPU= 59.205
  03:52:30 STEP  282 H= 282:00 +CPU= 95.660
  03:53:30 STEP  283 H= 283:00 +CPU= 59.890
  03:54:31 STEP  284 H= 284:00 +CPU= 60.962
  03:56:06 STEP  285 H= 285:00 +CPU= 94.844
  03:57:08 STEP  286 H= 286:00 +CPU= 61.460
  03:58:11 STEP  287 H= 287:00 +CPU= 62.717
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0379
(argv4) batchid: 945
(argv5) wuid: 12163468
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_278e7d643d0481c2e0330a6275697ecf to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
ic_ancil_file: ic_ancil_12163468
ifsdata_file: ifsdata_12163468
climate_data_file: clim_data_12163468
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_43854f33bbd5e4731ac504454f342a2c to: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd2d3009dbc0e0320f096bf80aa456aa to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481
The child process has been launched with process id: 267891
Executing the command: /var/lib/boinc-client/slots/4/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:267891:267891] [20221209:040318:1670554998:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:05:17 STEP  288 H= 288:00 +CPU=106.296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000288
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000288
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000276
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_0.zip
Uploading the intermediate file: upload_file_0.zip
Uploading trickle at timestep: 1036800
  04:06:19 STEP  289 H= 289:00 +CPU= 58.545
  04:07:22 STEP  290 H= 290:00 +CPU= 61.480
  04:09:01 STEP  291 H= 291:00 +CPU= 95.320
  04:10:02 STEP  292 H= 292:00 +CPU= 59.943
  04:10:59 STEP  293 H= 293:00 +CPU= 57.046
  04:12:40 STEP  294 H= 294:00 +CPU= 99.330
  04:13:42 STEP  295 H= 295:00 +CPU= 60.849
  04:14:42 STEP  296 H= 296:00 +CPU= 59.846
  04:16:19 STEP  297 H= 297:00 +CPU= 95.375
  04:17:20 STEP  298 H= 298:00 +CPU= 61.205
  04:18:22 STEP  299 H= 299:00 +CPU= 61.647
  04:20:10 STEP  300 H= 300:00 +CPU=105.656
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000300
  04:21:09 STEP  301 H= 301:00 +CPU= 58.548
  04:22:13 STEP  302 H= 302:00 +CPU= 62.598
  04:23:50 STEP  303 H= 303:00 +CPU= 94.901
  04:24:51 STEP  304 H= 304:00 +CPU= 59.801
  04:25:53 STEP  305 H= 305:00 +CPU= 60.974
  04:27:29 STEP  306 H= 306:00 +CPU= 94.820
  04:28:29 STEP  307 H= 307:00 +CPU= 59.181
  04:29:36 STEP  308 H= 308:00 +CPU= 63.656
  04:31:13 STEP  309 H= 309:00 +CPU= 95.492
  04:32:17 STEP  310 H= 310:00 +CPU= 62.397
  04:33:16 STEP  311 H= 311:00 +CPU= 58.918
  04:35:06 STEP  312 H= 312:00 +CPU=107.915
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000300
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 1123200
  04:36:08 STEP  313 H= 313:00 +CPU= 60.226
  04:37:12 STEP  314 H= 314:00 +CPU= 63.215
  04:38:53 STEP  315 H= 315:00 +CPU= 99.113
  04:39:56 STEP  316 H= 316:00 +CPU= 62.204
  04:40:57 STEP  317 H= 317:00 +CPU= 60.375
  04:42:37 STEP  318 H= 318:00 +CPU= 98.282
  04:43:36 STEP  319 H= 319:00 +CPU= 57.996
  04:44:36 STEP  320 H= 320:00 +CPU= 59.491
  04:46:12 STEP  321 H= 321:00 +CPU= 95.107
  04:47:15 STEP  322 H= 322:00 +CPU= 61.151
  04:48:14 STEP  323 H= 323:00 +CPU= 58.089
  04:49:58 STEP  324 H= 324:00 +CPU=101.842
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000324
  04:50:59 STEP  325 H= 325:00 +CPU= 60.561
  04:52:02 STEP  326 H= 326:00 +CPU= 62.284
  04:53:37 STEP  327 H= 327:00 +CPU= 93.673
  04:54:36 STEP  328 H= 328:00 +CPU= 59.525
  04:55:38 STEP  329 H= 329:00 +CPU= 60.400
  04:57:17 STEP  330 H= 330:00 +CPU= 96.894
  04:58:16 STEP  331 H= 331:00 +CPU= 58.701
  04:59:20 STEP  332 H= 332:00 +CPU= 63.154
  05:01:00 STEP  333 H= 333:00 +CPU= 97.965
  05:02:03 STEP  334 H= 334:00 +CPU= 60.773
  05:03:06 STEP  335 H= 335:00 +CPU= 61.368
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0379
(argv4) batchid: 945
(argv5) wuid: 12163468
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_278e7d643d0481c2e0330a6275697ecf to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
ic_ancil_file: ic_ancil_12163468
ifsdata_file: ifsdata_12163468
climate_data_file: clim_data_12163468
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_43854f33bbd5e4731ac504454f342a2c to: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd2d3009dbc0e0320f096bf80aa456aa to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481
The child process has been launched with process id: 268145
Executing the command: /var/lib/boinc-client/slots/4/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:268145:268145] [20221209:050905:1670558945:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:11:14 STEP  336 H= 336:00 +CPU=117.529
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000324
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_0.zip
Uploading the intermediate file: upload_file_0.zip
Uploading trickle at timestep: 1209600
  05:12:14 STEP  337 H= 337:00 +CPU= 58.889
  05:13:16 STEP  338 H= 338:00 +CPU= 60.206
  05:14:46 STEP  339 H= 339:00 +CPU= 89.957
  05:15:47 STEP  340 H= 340:00 +CPU= 60.032
  05:16:48 STEP  341 H= 341:00 +CPU= 60.222
  05:18:26 STEP  342 H= 342:00 +CPU= 96.669
  05:19:25 STEP  343 H= 343:00 +CPU= 58.177
  05:20:28 STEP  344 H= 344:00 +CPU= 62.766
  05:22:09 STEP  345 H= 345:00 +CPU= 98.294
  05:23:11 STEP  346 H= 346:00 +CPU= 61.133
  05:24:08 STEP  347 H= 347:00 +CPU= 56.800
  05:26:06 STEP  348 H= 348:00 +CPU=113.725
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000348
  05:27:03 STEP  349 H= 349:00 +CPU= 56.175
  05:28:04 STEP  350 H= 350:00 +CPU= 58.974
  05:29:41 STEP  351 H= 351:00 +CPU= 95.589
  05:30:41 STEP  352 H= 352:00 +CPU= 59.164
  05:31:41 STEP  353 H= 353:00 +CPU= 59.341
  05:33:23 STEP  354 H= 354:00 +CPU= 99.009
  06:19:29 STEP  355 H= 355:00 +CPU= 94.162
  06:20:30 STEP  356 H= 356:00 +CPU= 60.174
  06:22:10 STEP  357 H= 357:00 +CPU= 96.571
  06:23:12 STEP  358 H= 358:00 +CPU= 60.887
  06:24:16 STEP  359 H= 359:00 +CPU= 63.266
  06:26:18 STEP  360 H= 360:00 +CPU=116.381
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 1296000
  06:27:19 STEP  361 H= 361:00 +CPU= 60.342
  06:28:25 STEP  362 H= 362:00 +CPU= 64.126
  06:30:01 STEP  363 H= 363:00 +CPU= 94.935
  06:31:05 STEP  364 H= 364:00 +CPU= 62.472
  06:32:05 STEP  365 H= 365:00 +CPU= 59.503
  06:33:48 STEP  366 H= 366:00 +CPU=100.665
  06:34:50 STEP  367 H= 367:00 +CPU= 61.311
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0379
(argv4) batchid: 945
(argv5) wuid: 12163468
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_278e7d643d0481c2e0330a6275697ecf to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
ic_ancil_file: ic_ancil_12163468
ifsdata_file: ifsdata_12163468
climate_data_file: clim_data_12163468
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_43854f33bbd5e4731ac504454f342a2c to: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd2d3009dbc0e0320f096bf80aa456aa to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481
The child process has been launched with process id: 269013
Executing the command: /var/lib/boinc-client/slots/4/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269013:269013] [20221209:064014:1670564414:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:42:21 STEP  360 H= 360:00 +CPU=113.895
  06:43:21 STEP  361 H= 361:00 +CPU= 59.333
  06:44:21 STEP  362 H= 362:00 +CPU= 59.233
  06:46:02 STEP  363 H= 363:00 +CPU= 98.528
  06:47:06 STEP  364 H= 364:00 +CPU= 63.598
  06:48:07 STEP  365 H= 365:00 +CPU= 59.875
  06:49:44 STEP  366 H= 366:00 +CPU= 96.248
  06:50:42 STEP  367 H= 367:00 +CPU= 57.529
  06:51:47 STEP  368 H= 368:00 +CPU= 63.338
  06:53:31 STEP  369 H= 369:00 +CPU=103.615
  06:54:36 STEP  370 H= 370:00 +CPU= 64.241
  06:55:41 STEP  371 H= 371:00 +CPU= 64.087
  06:57:30 STEP  372 H= 372:00 +CPU=108.388
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000372
  06:58:31 STEP  373 H= 373:00 +CPU= 60.903
  06:59:35 STEP  374 H= 374:00 +CPU= 62.224
  07:01:09 STEP  375 H= 375:00 +CPU= 93.958
  07:02:11 STEP  376 H= 376:00 +CPU= 61.938
  07:03:11 STEP  377 H= 377:00 +CPU= 60.207
  07:04:48 STEP  378 H= 378:00 +CPU= 96.538
  07:05:49 STEP  379 H= 379:00 +CPU= 60.585
  07:06:52 STEP  380 H= 380:00 +CPU= 63.141
  07:08:28 STEP  381 H= 381:00 +CPU= 95.503
  07:09:30 STEP  382 H= 382:00 +CPU= 62.233
  07:10:33 STEP  383 H= 383:00 +CPU= 63.036
  07:12:35 STEP  384 H= 384:00 +CPU=118.189
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 1382400
  07:13:43 STEP  385 H= 385:00 +CPU= 66.942
  07:14:50 STEP  386 H= 386:00 +CPU= 65.251
  07:16:32 STEP  387 H= 387:00 +CPU=100.805
  07:17:37 STEP  388 H= 388:00 +CPU= 63.989
  07:18:37 STEP  389 H= 389:00 +CPU= 59.343
  07:20:17 STEP  390 H= 390:00 +CPU= 99.525
  07:21:20 STEP  391 H= 391:00 +CPU= 61.927
  07:22:23 STEP  392 H= 392:00 +CPU= 62.677
  07:24:02 STEP  393 H= 393:00 +CPU= 97.583
  07:25:09 STEP  394 H= 394:00 +CPU= 64.809
  07:26:09 STEP  395 H= 395:00 +CPU= 60.311
  07:27:55 STEP  396 H= 396:00 +CPU=104.924
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000396
  07:28:56 STEP  397 H= 397:00 +CPU= 59.353
  07:30:00 STEP  398 H= 398:00 +CPU= 63.046
  07:31:37 STEP  399 H= 399:00 +CPU= 96.350
  07:32:39 STEP  400 H= 400:00 +CPU= 60.858
  07:33:40 STEP  401 H= 401:00 +CPU= 59.807
  07:35:18 STEP  402 H= 402:00 +CPU= 96.087
  07:36:20 STEP  403 H= 403:00 +CPU= 60.795
  07:37:22 STEP  404 H= 404:00 +CPU= 60.701
  07:39:00 STEP  405 H= 405:00 +CPU= 95.610
  07:40:02 STEP  406 H= 406:00 +CPU= 61.337
  07:41:05 STEP  407 H= 407:00 +CPU= 61.480
  07:43:07 STEP  408 H= 408:00 +CPU=120.081
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000408
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000408
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_2.zip
Uploading the intermediate file: upload_file_2.zip
Uploading trickle at timestep: 1468800
  07:44:14 STEP  409 H= 409:00 +CPU= 64.625
  07:45:20 STEP  410 H= 410:00 +CPU= 65.371
  07:47:06 STEP  411 H= 411:00 +CPU=102.622
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0379
(argv4) batchid: 945
(argv5) wuid: 12163468
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_278e7d643d0481c2e0330a6275697ecf to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0379_2021050100_123_945_12163468.zip
ic_ancil_file: ic_ancil_12163468
ifsdata_file: ifsdata_12163468
climate_data_file: clim_data_12163468
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_43854f33bbd5e4731ac504454f342a2c to: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12163468.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd2d3009dbc0e0320f096bf80aa456aa to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12163468.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12163468.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481
The child process has been launched with process id: 269765
Executing the command: /var/lib/boinc-client/slots/4/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:B20:1:1:269765:269765] [20221209:075246:1670568766:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:54:53 STEP  408 H= 408:00 +CPU=115.838
  07:55:52 STEP  409 H= 409:00 +CPU= 58.303
  07:56:53 STEP  410 H= 410:00 +CPU= 58.242
  07:58:30 STEP  411 H= 411:00 +CPU= 94.529
  07:59:30 STEP  412 H= 412:00 +CPU= 59.990
  08:00:28 STEP  413 H= 413:00 +CPU= 57.242
  08:02:10 STEP  414 H= 414:00 +CPU=100.348
  08:03:08 STEP  415 H= 415:00 +CPU= 56.998
  08:04:08 STEP  416 H= 416:00 +CPU= 60.142
  08:05:46 STEP  417 H= 417:00 +CPU= 95.688
  08:06:46 STEP  418 H= 418:00 +CPU= 59.486
  08:07:43 STEP  419 H= 419:00 +CPU= 56.417
  08:09:29 STEP  420 H= 420:00 +CPU=104.469
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000420
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000420
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000420
  08:10:31 STEP  421 H= 421:00 +CPU= 60.422
  08:11:35 STEP  422 H= 422:00 +CPU= 62.464
  08:13:13 STEP  423 H= 423:00 +CPU= 96.675
  08:14:16 STEP  424 H= 424:00 +CPU= 61.444
  08:15:19 STEP  425 H= 425:00 +CPU= 60.679
  08:16:54 STEP  426 H= 426:00 +CPU= 93.503
  08:17:49 STEP  427 H= 427:00 +CPU= 55.082
  08:18:52 STEP  428 H= 428:00 +CPU= 62.169
  08:20:30 STEP  429 H= 429:00 +CPU= 94.947
  08:21:34 STEP  430 H= 430:00 +CPU= 63.072
  08:22:37 STEP  431 H= 431:00 +CPU= 61.380
  08:24:33 STEP  432 H= 432:00 +CPU=111.236
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000432
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000432
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMGGhpi1+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMSHhpi1+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163468/ICMUAhpi1+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0379_2021050100_123_945_12163468_1_r1719116481_1.zip
Uploading the intermediate file: upload_file_1.zip
Uploading trickle at timestep: 1555200
  08:25:36 STEP  433 H= 433:00 +CPU= 61.689
  08:26:40 STEP  434 H= 434:00 +CPU= 62.492
  08:28:22 STEP  435 H= 435:00 +CPU= 99.702
  08:29:26 STEP  436 H= 436:00 +CPU= 62.706
  08:30:26 STEP  437 H= 437:00 +CPU= 58.730
  08:32:07 STEP  438 H= 438:00 +CPU= 97.438
  08:33:07 STEP  439 H= 439:00 +CPU= 59.330
  08:34:14 STEP  440 H= 440:00 +CPU= 65.447
  08:35:49 STEP  441 H= 441:00 +CPU= 93.413
  08:36:51 STEP  442 H= 442:00 +CPU= 61.320
  08:37:58 STEP  443 H= 443:00 +CPU= 63.706
  08:39:48 STEP  444 H= 444:00 +CPU=106.744
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhpi1+000444
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhpi1+000444
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhpi1+000444
  08:40:53 STEP  445 H= 445:00 +CPU= 63.401
  08:41:56 STEP  446 H= 446:00 +CPU= 62.269
  08:43:36 STEP  447 H= 447:00 +CPU= 95.982
  08:44:40 STEP  448 H= 448:00 +CPU= 63.208
  08:45:41 STEP  449 H= 449:00 +CPU= 60.239
  08:47:21 STEP  450 H= 450:00 +CPU= 98.918
  08:48:22 STEP  451 H= 451:00 +CPU= 60.117
forrtl: No space left on device
forrtl: severe (38): error during write, unit 20, file /var/lib/boinc-client/slots/4/NODE.001_01
Image              PC                Routine            Line        Source             
master.exe         0000000001CD9519  Unknown               Unknown  Unknown
master.exe         0000000001D45366  Unknown               Unknown  Unknown
master.exe         0000000001D408EE  Unknown               Unknown  Unknown
master.exe         0000000000873991  wavemdl_                  840  wavemdl.F
master.exe         00000000007E0165  wvcouple_                 260  wvcouple.F90
master.exe         000000000055E4AD  cnt4_                    1283  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:38:04 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,555,200 1,868 0.0012
09 May 2023 15:38:04 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,468,800 3,724 0.0025
09 May 2023 15:38:01 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,382,400 1,925 0.0014
09 May 2023 15:37:59 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,296,000 1,927 0.0015
09 May 2023 15:37:57 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,209,600 125 0.0001
09 May 2023 15:37:57 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,123,200 1,875 0.0017
09 May 2023 15:37:55 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 1,036,800 114 0.0001
09 May 2023 15:37:54 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 950,400 1,929 0.0020
09 May 2023 15:37:50 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 864,000 1,923 0.0022
09 May 2023 15:37:50 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 777,600 3,688 0.0047
09 May 2023 15:37:48 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 691,200 1,896 0.0027
09 May 2023 15:37:46 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 604,800 3,642 0.0060
09 May 2023 15:37:46 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 518,400 1,891 0.0036
09 May 2023 15:37:44 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 432,000 3,661 0.0085
09 May 2023 15:37:44 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 345,600 1,887 0.0055
09 May 2023 15:37:42 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 259,200 3,664 0.0141
09 May 2023 15:37:40 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 172,800 1,892 0.0109
09 May 2023 15:37:36 1536378 22246932 oifs_43r3_ps_0379_2021050100_123_945_12163468_1 86,400 1,892 0.0219


©2024 climateprediction.net