<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0990
(argv4) batchid: 976
(argv5) wuid: 12193634
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /config/slots/1
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_ps_12193634
Copying: /config/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3ca2485c1eb180929769459f863ed344 to: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
Unzipping the namelist zip file: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
ic_ancil_file: ic_ancil_12193634
ifsdata_file: ifsdata_12193634
climate_data_file: clim_data_12193634
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3c4b804d7040d2ed6266e093bd33461f to: /config/slots/1/ic_ancil_12193634.zip
Unzipping the IC ancils zip file: /config/slots/1/ic_ancil_12193634.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1e453926f33dbda43f0195bdbcd61a8e to: /config/slots/1/ifsdata/ifsdata_12193634.zip
Unzipping the ifsdata_zip file: /config/slots/1/ifsdata/ifsdata_12193634.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/1/159l_2/clim_data_12193634.zip
Unzipping the climate data zip file: /config/slots/1/159l_2/clim_data_12193634.zip
Checking for progress XML file: /config/slots/1/progress_file_12193634.xml
Creating progress file: /config/slots/1/progress_file_12193634.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515
The child process has been launched with process id: 736
Executing the command: /config/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:736:736] [20221225:203339:1672000419:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
20:34:09 STEP 0 H= 0:00 +CPU= 27.798
Moving to projects directory: /config/slots/1/ICMGGhq0f+000000
Moving to projects directory: /config/slots/1/ICMSHhq0f+000000
Moving to projects directory: /config/slots/1/ICMUAhq0f+000000
20:34:27 STEP 1 H= 1:00 +CPU= 17.977
20:34:44 STEP 2 H= 2:00 +CPU= 16.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:35:37 STEP 3 H= 3:00 +CPU= 26.153
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:36:13 STEP 4 H= 4:00 +CPU= 17.094
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:36:45 STEP 5 H= 5:00 +CPU= 15.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:37:23 STEP 6 H= 6:00 +CPU= 25.407
20:37:38 STEP 7 H= 7:00 +CPU= 14.988
20:37:54 STEP 8 H= 8:00 +CPU= 15.616
20:38:21 STEP 9 H= 9:00 +CPU= 26.517
20:38:39 STEP 10 H= 10:00 +CPU= 16.970
20:38:56 STEP 11 H= 11:00 +CPU= 16.480
20:39:23 STEP 12 H= 12:00 +CPU= 27.129
Moving to projects directory: /config/slots/1/ICMGGhq0f+000012
Moving to projects directory: /config/slots/1/ICMSHhq0f+000012
Moving to projects directory: /config/slots/1/ICMUAhq0f+000012
20:39:40 STEP 13 H= 13:00 +CPU= 16.098
20:39:57 STEP 14 H= 14:00 +CPU= 16.975
20:40:22 STEP 15 H= 15:00 +CPU= 24.410
20:40:37 STEP 16 H= 16:00 +CPU= 15.134
20:40:53 STEP 17 H= 17:00 +CPU= 15.300
20:41:16 STEP 18 H= 18:00 +CPU= 23.866
20:41:31 STEP 19 H= 19:00 +CPU= 14.639
20:41:46 STEP 20 H= 20:00 +CPU= 14.879
20:42:09 STEP 21 H= 21:00 +CPU= 22.925
20:42:24 STEP 22 H= 22:00 +CPU= 14.841
20:42:39 STEP 23 H= 23:00 +CPU= 14.824
20:43:06 STEP 24 H= 24:00 +CPU= 27.073
Moving to projects directory: /config/slots/1/ICMGGhq0f+000024
Moving to projects directory: /config/slots/1/ICMSHhq0f+000024
Moving to projects directory: /config/slots/1/ICMUAhq0f+000024
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000000
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000000
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000000
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000012
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000012
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000012
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_0.zip
Uploading the intermediate file: upload_file_0.zip
20:43:21 STEP 25 H= 25:00 +CPU= 14.462
20:43:36 STEP 26 H= 26:00 +CPU= 14.950
20:43:59 STEP 27 H= 27:00 +CPU= 22.838
20:44:14 STEP 28 H= 28:00 +CPU= 15.009
20:44:28 STEP 29 H= 29:00 +CPU= 14.483
20:44:52 STEP 30 H= 30:00 +CPU= 23.397
20:45:06 STEP 31 H= 31:00 +CPU= 14.446
20:45:21 STEP 32 H= 32:00 +CPU= 14.993
20:45:44 STEP 33 H= 33:00 +CPU= 22.988
20:45:59 STEP 34 H= 34:00 +CPU= 14.986
20:46:14 STEP 35 H= 35:00 +CPU= 14.706
20:46:40 STEP 36 H= 36:00 +CPU= 25.407
Moving to projects directory: /config/slots/1/ICMGGhq0f+000036
Moving to projects directory: /config/slots/1/ICMSHhq0f+000036
Moving to projects directory: /config/slots/1/ICMUAhq0f+000036
20:46:54 STEP 37 H= 37:00 +CPU= 14.464
20:47:09 STEP 38 H= 38:00 +CPU= 14.968
20:47:32 STEP 39 H= 39:00 +CPU= 22.937
20:47:47 STEP 40 H= 40:00 +CPU= 14.806
20:48:01 STEP 41 H= 41:00 +CPU= 14.405
20:48:25 STEP 42 H= 42:00 +CPU= 23.365
20:48:39 STEP 43 H= 43:00 +CPU= 14.460
20:48:54 STEP 44 H= 44:00 +CPU= 14.957
20:49:17 STEP 45 H= 45:00 +CPU= 22.833
20:49:32 STEP 46 H= 46:00 +CPU= 14.797
20:49:47 STEP 47 H= 47:00 +CPU= 14.767
20:50:14 STEP 48 H= 48:00 +CPU= 27.101
Moving to projects directory: /config/slots/1/ICMGGhq0f+000048
Moving to projects directory: /config/slots/1/ICMSHhq0f+000048
Moving to projects directory: /config/slots/1/ICMUAhq0f+000048
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000024
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000024
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000024
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000036
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000036
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000036
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_1.zip
Uploading the intermediate file: upload_file_1.zip
20:50:29 STEP 49 H= 49:00 +CPU= 14.355
20:50:43 STEP 50 H= 50:00 +CPU= 14.796
20:51:06 STEP 51 H= 51:00 +CPU= 22.783
20:51:21 STEP 52 H= 52:00 +CPU= 14.889
20:51:36 STEP 53 H= 53:00 +CPU= 14.481
20:51:59 STEP 54 H= 54:00 +CPU= 23.349
20:52:14 STEP 55 H= 55:00 +CPU= 14.512
20:52:29 STEP 56 H= 56:00 +CPU= 14.943
20:52:52 STEP 57 H= 57:00 +CPU= 22.827
20:53:06 STEP 58 H= 58:00 +CPU= 14.821
20:53:21 STEP 59 H= 59:00 +CPU= 14.461
20:53:46 STEP 60 H= 60:00 +CPU= 25.261
Moving to projects directory: /config/slots/1/ICMGGhq0f+000060
Moving to projects directory: /config/slots/1/ICMSHhq0f+000060
Moving to projects directory: /config/slots/1/ICMUAhq0f+000060
20:54:01 STEP 61 H= 61:00 +CPU= 14.420
20:54:16 STEP 62 H= 62:00 +CPU= 14.947
20:54:39 STEP 63 H= 63:00 +CPU= 22.848
20:54:53 STEP 64 H= 64:00 +CPU= 14.843
20:55:08 STEP 65 H= 65:00 +CPU= 14.429
20:55:31 STEP 66 H= 66:00 +CPU= 23.392
20:55:46 STEP 67 H= 67:00 +CPU= 14.379
20:56:01 STEP 68 H= 68:00 +CPU= 14.837
20:56:24 STEP 69 H= 69:00 +CPU= 22.762
20:56:38 STEP 70 H= 70:00 +CPU= 14.881
20:56:53 STEP 71 H= 71:00 +CPU= 14.818
20:57:21 STEP 72 H= 72:00 +CPU= 26.996
Moving to projects directory: /config/slots/1/ICMGGhq0f+000072
Moving to projects directory: /config/slots/1/ICMSHhq0f+000072
Moving to projects directory: /config/slots/1/ICMUAhq0f+000072
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000048
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000048
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000048
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000060
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000060
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000060
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_2.zip
Uploading the intermediate file: upload_file_2.zip
20:57:35 STEP 73 H= 73:00 +CPU= 14.603
20:57:50 STEP 74 H= 74:00 +CPU= 14.881
20:58:13 STEP 75 H= 75:00 +CPU= 22.963
20:58:28 STEP 76 H= 76:00 +CPU= 14.781
20:58:42 STEP 77 H= 77:00 +CPU= 14.369
20:59:06 STEP 78 H= 78:00 +CPU= 23.396
20:59:20 STEP 79 H= 79:00 +CPU= 14.418
20:59:35 STEP 80 H= 80:00 +CPU= 15.030
20:59:58 STEP 81 H= 81:00 +CPU= 22.790
21:00:13 STEP 82 H= 82:00 +CPU= 14.969
21:00:28 STEP 83 H= 83:00 +CPU= 14.449
21:00:53 STEP 84 H= 84:00 +CPU= 25.440
Moving to projects directory: /config/slots/1/ICMGGhq0f+000084
Moving to projects directory: /config/slots/1/ICMSHhq0f+000084
Moving to projects directory: /config/slots/1/ICMUAhq0f+000084
21:01:08 STEP 85 H= 85:00 +CPU= 14.612
21:01:23 STEP 86 H= 86:00 +CPU= 14.879
21:01:46 STEP 87 H= 87:00 +CPU= 22.810
21:02:01 STEP 88 H= 88:00 +CPU= 14.906
21:02:15 STEP 89 H= 89:00 +CPU= 14.613
21:02:39 STEP 90 H= 90:00 +CPU= 23.453
21:02:53 STEP 91 H= 91:00 +CPU= 14.409
21:03:08 STEP 92 H= 92:00 +CPU= 14.901
21:03:31 STEP 93 H= 93:00 +CPU= 22.894
21:03:46 STEP 94 H= 94:00 +CPU= 14.928
21:04:01 STEP 95 H= 95:00 +CPU= 14.853
21:04:28 STEP 96 H= 96:00 +CPU= 27.113
Moving to projects directory: /config/slots/1/ICMGGhq0f+000096
Moving to projects directory: /config/slots/1/ICMSHhq0f+000096
Moving to projects directory: /config/slots/1/ICMUAhq0f+000096
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000072
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000072
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000072
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000084
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000084
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000084
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_3.zip
Uploading the intermediate file: upload_file_3.zip
21:04:43 STEP 97 H= 97:00 +CPU= 14.370
21:04:58 STEP 98 H= 98:00 +CPU= 14.785
21:05:21 STEP 99 H= 99:00 +CPU= 22.875
21:05:36 STEP 100 H= 100:00 +CPU= 14.875
21:05:50 STEP 101 H= 101:00 +CPU= 14.477
21:06:13 STEP 102 H= 102:00 +CPU= 23.323
21:06:28 STEP 103 H= 103:00 +CPU= 14.473
21:06:43 STEP 104 H= 104:00 +CPU= 14.931
21:07:06 STEP 105 H= 105:00 +CPU= 22.809
21:07:21 STEP 106 H= 106:00 +CPU= 14.973
21:07:35 STEP 107 H= 107:00 +CPU= 14.566
21:08:01 STEP 108 H= 108:00 +CPU= 25.356
Moving to projects directory: /config/slots/1/ICMGGhq0f+000108
Moving to projects directory: /config/slots/1/ICMSHhq0f+000108
Moving to projects directory: /config/slots/1/ICMUAhq0f+000108
21:08:15 STEP 109 H= 109:00 +CPU= 14.443
21:08:30 STEP 110 H= 110:00 +CPU= 14.909
21:08:53 STEP 111 H= 111:00 +CPU= 22.872
21:09:08 STEP 112 H= 112:00 +CPU= 14.897
21:09:23 STEP 113 H= 113:00 +CPU= 14.484
21:09:46 STEP 114 H= 114:00 +CPU= 23.201
21:10:00 STEP 115 H= 115:00 +CPU= 14.344
21:10:15 STEP 116 H= 116:00 +CPU= 14.927
21:10:38 STEP 117 H= 117:00 +CPU= 22.814
21:10:53 STEP 118 H= 118:00 +CPU= 14.938
21:11:08 STEP 119 H= 119:00 +CPU= 14.860
21:11:35 STEP 120 H= 120:00 +CPU= 27.024
Moving to projects directory: /config/slots/1/ICMGGhq0f+000120
Moving to projects directory: /config/slots/1/ICMSHhq0f+000120
Moving to projects directory: /config/slots/1/ICMUAhq0f+000120
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000096
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000096
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000096
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000108
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000108
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000108
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_4.zip
Uploading the intermediate file: upload_file_4.zip
21:11:50 STEP 121 H= 121:00 +CPU= 14.460
21:12:05 STEP 122 H= 122:00 +CPU= 14.896
21:12:28 STEP 123 H= 123:00 +CPU= 22.783
21:12:43 STEP 124 H= 124:00 +CPU= 14.828
21:12:57 STEP 125 H= 125:00 +CPU= 14.554
21:13:21 STEP 126 H= 126:00 +CPU= 23.332
21:13:35 STEP 127 H= 127:00 +CPU= 14.452
21:13:50 STEP 128 H= 128:00 +CPU= 15.017
21:14:13 STEP 129 H= 129:00 +CPU= 22.820
21:14:28 STEP 130 H= 130:00 +CPU= 14.952
21:14:42 STEP 131 H= 131:00 +CPU= 14.343
21:15:08 STEP 132 H= 132:00 +CPU= 25.341
Moving to projects directory: /config/slots/1/ICMGGhq0f+000132
Moving to projects directory: /config/slots/1/ICMSHhq0f+000132
Moving to projects directory: /config/slots/1/ICMUAhq0f+000132
21:15:22 STEP 133 H= 133:00 +CPU= 14.573
21:15:37 STEP 134 H= 134:00 +CPU= 14.878
21:16:00 STEP 135 H= 135:00 +CPU= 22.816
21:16:15 STEP 136 H= 136:00 +CPU= 14.959
21:16:30 STEP 137 H= 137:00 +CPU= 14.419
21:16:53 STEP 138 H= 138:00 +CPU= 23.231
21:17:07 STEP 139 H= 139:00 +CPU= 14.392
21:17:22 STEP 140 H= 140:00 +CPU= 15.008
21:17:45 STEP 141 H= 141:00 +CPU= 22.885
21:18:00 STEP 142 H= 142:00 +CPU= 14.804
21:18:15 STEP 143 H= 143:00 +CPU= 14.928
21:18:43 STEP 144 H= 144:00 +CPU= 26.954
Moving to projects directory: /config/slots/1/ICMGGhq0f+000144
Moving to projects directory: /config/slots/1/ICMSHhq0f+000144
Moving to projects directory: /config/slots/1/ICMUAhq0f+000144
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000120
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000120
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000120
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000132
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000132
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000132
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_5.zip
Uploading the intermediate file: upload_file_5.zip
21:18:57 STEP 145 H= 145:00 +CPU= 14.409
21:19:12 STEP 146 H= 146:00 +CPU= 14.998
21:19:35 STEP 147 H= 147:00 +CPU= 22.942
21:19:50 STEP 148 H= 148:00 +CPU= 14.862
21:20:05 STEP 149 H= 149:00 +CPU= 14.427
21:20:28 STEP 150 H= 150:00 +CPU= 23.324
21:20:42 STEP 151 H= 151:00 +CPU= 14.313
21:20:57 STEP 152 H= 152:00 +CPU= 14.945
21:21:20 STEP 153 H= 153:00 +CPU= 22.898
21:21:35 STEP 154 H= 154:00 +CPU= 14.844
21:21:50 STEP 155 H= 155:00 +CPU= 14.577
21:22:15 STEP 156 H= 156:00 +CPU= 25.349
Moving to projects directory: /config/slots/1/ICMGGhq0f+000156
Moving to projects directory: /config/slots/1/ICMSHhq0f+000156
Moving to projects directory: /config/slots/1/ICMUAhq0f+000156
21:22:30 STEP 157 H= 157:00 +CPU= 14.498
21:22:45 STEP 158 H= 158:00 +CPU= 14.944
21:23:08 STEP 159 H= 159:00 +CPU= 22.879
21:23:22 STEP 160 H= 160:00 +CPU= 14.855
21:23:37 STEP 161 H= 161:00 +CPU= 14.328
21:24:00 STEP 162 H= 162:00 +CPU= 23.441
21:24:15 STEP 163 H= 163:00 +CPU= 14.527
21:24:30 STEP 164 H= 164:00 +CPU= 14.874
21:24:53 STEP 165 H= 165:00 +CPU= 22.926
21:25:08 STEP 166 H= 166:00 +CPU= 14.848
21:25:23 STEP 167 H= 167:00 +CPU= 14.921
21:25:50 STEP 168 H= 168:00 +CPU= 26.993
Moving to projects directory: /config/slots/1/ICMGGhq0f+000168
Moving to projects directory: /config/slots/1/ICMSHhq0f+000168
Moving to projects directory: /config/slots/1/ICMUAhq0f+000168
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000144
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000144
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000144
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMGGhq0f+000156
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMSHhq0f+000156
Adding to the zip: /config/projects/climateprediction.net/oifs_43r3_ps_12193634/ICMUAhq0f+000156
Zipping up the intermediate file: /config/projects/climateprediction.net/oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515_6.zip
Uploading the intermediate file: upload_file_6.zip
21:26:04 STEP 169 H= 169:00 +CPU= 14.439
21:26:19 STEP 170 H= 170:00 +CPU= 14.964
21:26:42 STEP 171 H= 171:00 +CPU= 22.782
21:26:57 STEP 172 H= 172:00 +CPU= 14.890
21:27:12 STEP 173 H= 173:00 +CPU= 14.437
21:27:35 STEP 174 H= 174:00 +CPU= 23.315
21:27:49 STEP 175 H= 175:00 +CPU= 14.537
21:28:05 STEP 176 H= 176:00 +CPU= 14.983
21:28:27 STEP 177 H= 177:00 +CPU= 22.901
21:28:42 STEP 178 H= 178:00 +CPU= 14.979
21:28:57 STEP 179 H= 179:00 +CPU= 14.451
21:29:22 STEP 180 H= 180:00 +CPU= 25.182
Moving to projects directory: /config/slots/1/ICMGGhq0f+000180
Moving to projects directory: /config/slots/1/ICMSHhq0f+000180
Moving to projects directory: /config/slots/1/ICMUAhq0f+000180
21:29:37 STEP 181 H= 181:00 +CPU= 14.458
21:29:52 STEP 182 H= 182:00 +CPU= 14.876
21:30:15 STEP 183 H= 183:00 +CPU= 22.886
21:30:29 STEP 184 H= 184:00 +CPU= 14.764
21:30:44 STEP 185 H= 185:00 +CPU= 14.658
21:31:14 STEP 186 H= 186:00 +CPU= 27.214
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0990
(argv4) batchid: 976
(argv5) wuid: 12193634
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /config/slots/1
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_ps_12193634
..mkdir for temp folder for results failed
Copying: /config/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3ca2485c1eb180929769459f863ed344 to: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
Unzipping the namelist zip file: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
ic_ancil_file: ic_ancil_12193634
ifsdata_file: ifsdata_12193634
climate_data_file: clim_data_12193634
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3c4b804d7040d2ed6266e093bd33461f to: /config/slots/1/ic_ancil_12193634.zip
Unzipping the IC ancils zip file: /config/slots/1/ic_ancil_12193634.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1e453926f33dbda43f0195bdbcd61a8e to: /config/slots/1/ifsdata/ifsdata_12193634.zip
Unzipping the ifsdata_zip file: /config/slots/1/ifsdata/ifsdata_12193634.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/1/159l_2/clim_data_12193634.zip
Unzipping the climate data zip file: /config/slots/1/159l_2/clim_data_12193634.zip
Checking for progress XML file: /config/slots/1/progress_file_12193634.xml
Opened progress file ok : /config/slots/1/progress_file_12193634.xml
-- Model is restarting --
Adjusting last_iter, 187, to previous model restart step.
Creating progress file: /config/slots/1/progress_file_12193634.xml
last_cpu_time: 3364
upload_file_number: 7
last_iter: 167
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515
The child process has been launched with process id: 3870
Executing the command: /config/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:3870:3870] [20221225:213632:1672004192:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:37:29 STEP 168 H= 168:00 +CPU= 46.362
21:38:01 STEP 169 H= 169:00 +CPU= 22.119
21:38:55 STEP 170 H= 170:00 +CPU= 26.318
21:39:49 STEP 171 H= 171:00 +CPU= 42.078
21:40:18 STEP 172 H= 172:00 +CPU= 27.238
21:40:50 STEP 173 H= 173:00 +CPU= 28.755
21:42:08 STEP 174 H= 174:00 +CPU= 41.588
21:42:53 STEP 175 H= 175:00 +CPU= 27.347
21:43:24 STEP 176 H= 176:00 +CPU= 29.272
21:44:09 STEP 177 H= 177:00 +CPU= 39.519
21:44:43 STEP 178 H= 178:00 +CPU= 30.132
21:45:17 STEP 179 H= 179:00 +CPU= 23.616
21:46:43 STEP 180 H= 180:00 +CPU= 45.415
Moving to projects directory: /config/slots/1/ICMGGhq0f+000180
Moving to projects directory: /config/slots/1/ICMSHhq0f+000180
Moving to projects directory: /config/slots/1/ICMUAhq0f+000180
21:47:15 STEP 181 H= 181:00 +CPU= 28.811
21:47:46 STEP 182 H= 182:00 +CPU= 25.725
21:48:36 STEP 183 H= 183:00 +CPU= 44.907
22:09:43 STEP 184 H= 184:00 +CPU= 32.221
22:10:23 STEP 185 H= 185:00 +CPU= 29.201
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0990
(argv4) batchid: 976
(argv5) wuid: 12193634
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /config/slots/1
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_ps_12193634
..mkdir for temp folder for results failed
Copying: /config/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3ca2485c1eb180929769459f863ed344 to: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
Unzipping the namelist zip file: /config/slots/1/oifs_43r3_ps_0990_2007050100_123_976_12193634.zip
ic_ancil_file: ic_ancil_12193634
ifsdata_file: ifsdata_12193634
climate_data_file: clim_data_12193634
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3c4b804d7040d2ed6266e093bd33461f to: /config/slots/1/ic_ancil_12193634.zip
Unzipping the IC ancils zip file: /config/slots/1/ic_ancil_12193634.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1e453926f33dbda43f0195bdbcd61a8e to: /config/slots/1/ifsdata/ifsdata_12193634.zip
Unzipping the ifsdata_zip file: /config/slots/1/ifsdata/ifsdata_12193634.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/1/159l_2/clim_data_12193634.zip
Unzipping the climate data zip file: /config/slots/1/159l_2/clim_data_12193634.zip
Checking for progress XML file: /config/slots/1/progress_file_12193634.xml
Opened progress file ok : /config/slots/1/progress_file_12193634.xml
-- Model is restarting --
Adjusting last_iter, 186, to previous model restart step.
Creating progress file: /config/slots/1/progress_file_12193634.xml
last_cpu_time: 3959
upload_file_number: 7
last_iter: 167
last_upload: 604800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0990_2007050100_123_976_12193634_0_r559592515
The child process has been launched with process id: 4668
Executing the command: /config/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:5c4ac435be4b:1:1:4668:4668] [20221225:221140:1672006300:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
22:12:49 STEP 168 H= 168:00 +CPU= 47.562
22:13:27 STEP 169 H= 169:00 +CPU= 28.850
22:14:05 STEP 170 H= 170:00 +CPU= 26.110
22:14:47 STEP 171 H= 171:00 +CPU= 40.100
22:15:26 STEP 172 H= 172:00 +CPU= 24.429
22:16:04 STEP 173 H= 173:00 +CPU= 31.057
22:16:59 STEP 174 H= 174:00 +CPU= 39.684
22:17:46 STEP 175 H= 175:00 +CPU= 28.328
22:18:14 STEP 176 H= 176:00 +CPU= 27.050
22:19:10 STEP 177 H= 177:00 +CPU= 38.467
22:19:42 STEP 178 H= 178:00 +CPU= 28.773
22:20:12 STEP 179 H= 179:00 +CPU= 26.044
22:20:57 STEP 180 H= 180:00 +CPU= 41.628
Moving to projects directory: /config/slots/1/ICMGGhq0f+000180
Moving to projects directory: /config/slots/1/ICMSHhq0f+000180
Moving to projects directory: /config/slots/1/ICMUAhq0f+000180
22:21:30 STEP 181 H= 181:00 +CPU= 25.809
22:22:06 STEP 182 H= 182:00 +CPU= 30.340
22:22:54 STEP 183 H= 183:00 +CPU= 42.895
22:24:03 STEP 184 H= 184:00 +CPU= 29.179
22:24:33 STEP 185 H= 185:00 +CPU= 28.211
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
