<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:49988:49988] [20240614:120205:1718359325:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:07:03 STEP 0 H= 0:00 +CPU= 22.508
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 1, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 34
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 10239
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10239:10239] [20240614:162424:1718375064:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 42
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 33780
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:33780:33780] [20240614:190948:1718384988:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:10:59 STEP 0 H= 0:00 +CPU= 20.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
19:11:14 STEP 1 H= 0:15 +CPU= 11.459
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
19:11:24 STEP 2 H= 0:30 +CPU= 9.753
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:11:45 STEP 3 H= 0:45 +CPU= 9.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 1, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 72
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 41447
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:41447:41447] [20240614:191543:1718385343:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 0, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 87
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 10166
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:10166:10166] [20240614:192458:1718385898:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:25:34 STEP 0 H= 0:00 +CPU= 14.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /home/i3/slots/3/ICMGGh7zg+000000
Moving to projects directory: /home/i3/slots/3/ICMSHh7zg+000000
19:25:50 STEP 1 H= 0:15 +CPU= 13.140
Moving to projects directory: /home/i3/slots/3/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:26:08 STEP 2 H= 0:30 +CPU= 13.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
19:26:22 STEP 3 H= 0:45 +CPU= 12.455
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3dl
(argv4) batchid: 1018
(argv5) wuid: 12292704
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/i3/slots/3
Project directory is: /home/i3/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/i3/projects/climateprediction.net/oifs_43r3_bl_12292704
..mkdir for temp folder for results failed
Copying: /home/i3/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/i3/slots/3/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b23ff273b95b2d3d49265b2d6d178b19 to: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
Unzipping the namelist zip file: /home/i3/slots/3/oifs_43r3_bl_a3dl_2016092300_20_1018_12292704.zip
ic_ancil_file: ic_ancil_12292704
ifsdata_file: ifsdata_12292704
climate_data_file: clim_data_12292704
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/i3/slots/3/ic_ancil_12292704.zip
Unzipping the IC ancils zip file: /home/i3/slots/3/ic_ancil_12292704.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e1c37b5d8581da24b85a4837aa9564a8 to: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
Unzipping the ifsdata_zip file: /home/i3/slots/3/ifsdata/ifsdata_12292704.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Unzipping the climate data zip file: /home/i3/slots/3/159l_2/clim_data_12292704.zip
Checking for progress XML file: /home/i3/slots/3/progress_file_12292704.xml
Opened progress file ok : /home/i3/slots/3/progress_file_12292704.xml
-- Model is restarting --
Adjusting last_iter, 4, to previous model restart step.
Creating progress file: /home/i3/slots/3/progress_file_12292704.xml
last_cpu_time: 150
upload_file_number: 0
last_iter: 1
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3dl_2016092300_20_1018_12292704_0_r1920888991
The child process has been launched with process id: 11674
Executing the command: /home/i3/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.117] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:PIIPV2-AM4:1:1:11674:11674] [20240614:192858:1718386138:0.171] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc
UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED
SUECAEC: INLEV,ISYEAR,IEYEAR= 0 1957 2010
suecaec NAERMACC = 1
UPDT RAESHDU = 2000.0
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20160923
90 9.999999999999998E-005
DATE= 2016 9 23
NORMS AT NSTEP CNT4 0
SPECTRAL NORMS - LOG(PREHYDS) 0.115260884514965E+02
LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY
AVE 0.866154844891897E-06 0.555153323095658E-07 0.217247615887419E+03 0.180297554151816E+01
GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM
AVE 0.863401149692152E-03 0.000000000000000E+00 0.516457675471079E-02
GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
NSTEP = 0 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 360415332 BYTES IN 6 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.4473( 0.0%)
NSTEP = 0 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000000
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0142 SECONDS 49816.8 Mbytes/s, TOTAL TIME= 0.2388( 5.9%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 4939338816 BYTES IN 125 RECORDS TRANSFERRED IN 0.0049 SECONDS******** Mbytes/s, TOTAL TIME= 0.1145( 4.3%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0002 SECONDS 0.0 Mbytes/s, TOTAL TIME= 0.0046( 4.3%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 300128947 BYTES IN 4 RECORDS TRANSFERRED IN 0.0005 SECONDS600257.9 Mbytes/s, TOTAL TIME= 0.0056( 8.9%)
STEPO_FPOS WAS : T00000000M
NSTEP = 0 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 7 lines from file: ifs.stat
19:29:39 000000000 CNT3 -999 0.371 0.371 38.939 0:00 0:00 0.00000000000000E+00 1GB 0MB
19:29:47 A00000000 STEPO 0 4.677 4.677 46.599 0:04 0:47 0.55515332309566E-07 2GB 0MB
19:29:49 0AA000000 STEPO 0 0.058 0.058 2.252 0:04 0:49 0.55515332309566E-07 2GB 0MB
19:29:50 FULLPOS-B DYNFPOS 0 0.788 0.788 1.489 0:05 0:50 0.55515332309566E-07 2GB 0MB
19:29:55 FULLPOS-V DYNFPOS 0 2.360 2.360 4.976 0:07 0:55 0.55515332309566E-07 2GB 0MB
19:29:56 FULLPOS-T DYNFPOS 0 0.470 0.470 0.470 0:08 0:56 0.55515332309566E-07 2GB 0MB
19:29:56 0AAA00AAA STEPO 0 0.347 0.347 0.347 0:08 0:56 0.55515332309566E-07 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /home/i3/slots/3/progress_file_12292704.xml
<running_values>
<last_cpu_time>156.700000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>4</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
