<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a4jp
(argv4) batchid: 1017
(argv5) wuid: 12287720
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /config/slots/5
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_bl_12287720
Copying: /config/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /config/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/5/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2d22e8108368ba5b6b8576fb47c57c7c to: /config/slots/5/oifs_43r3_bl_a4jp_2016092300_20_1017_12287720.zip
Unzipping the namelist zip file: /config/slots/5/oifs_43r3_bl_a4jp_2016092300_20_1017_12287720.zip
ic_ancil_file: ic_ancil_12287720
ifsdata_file: ifsdata_12287720
climate_data_file: clim_data_12287720
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /config/slots/5/ic_ancil_12287720.zip
Unzipping the IC ancils zip file: /config/slots/5/ic_ancil_12287720.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6a7a3278b6d577c48743354284b9c33 to: /config/slots/5/ifsdata/ifsdata_12287720.zip
Unzipping the ifsdata_zip file: /config/slots/5/ifsdata/ifsdata_12287720.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/5/159l_2/clim_data_12287720.zip
Unzipping the climate data zip file: /config/slots/5/159l_2/clim_data_12287720.zip
Checking for progress XML file: /config/slots/5/progress_file_12287720.xml
Creating progress file: /config/slots/5/progress_file_12287720.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a4jp_2016092300_20_1017_12287720_1_r938434124
The child process has been launched with process id: 5922
Executing the command: /config/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:boinc-648c58ffd-75q9k:1:1:5922:5922] [20240608:043929:1717835969:0.002] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
04:40:48 STEP 0 H= 0:00 +CPU= 28.759
Moving to projects directory: /config/slots/5/ICMGGh7zg+000000
Moving to projects directory: /config/slots/5/ICMSHh7zg+000000
Moving to projects directory: /config/slots/5/ICMUAh7zg+000000
04:41:10 STEP 1 H= 0:15 +CPU= 21.957
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
AVE 0.282099638049642E-05 0.555153323095658E-07 0.209452777300736E+03 0.185949380131754E+02
GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM
AVE 0.112719223335735E-02 0.000000000000000E+00 0.738254592731970E-02
GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
NSTEP = 0 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 361192964 BYTES IN 6 RECORDS TRANSFERRED IN 0.0004 SECONDS902982.4 Mbytes/s, TOTAL TIME= 0.0502( 0.8%)
NSTEP = 0 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000000
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0051 SECONDS138705.7 Mbytes/s, TOTAL TIME= 0.2712( 1.9%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 4951780928 BYTES IN 125 RECORDS TRANSFERRED IN 0.0020 SECONDS******** Mbytes/s, TOTAL TIME= 0.1226( 1.6%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0032( 3.1%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 300906579 BYTES IN 4 RECORDS TRANSFERRED IN 0.0001 SECONDS******** Mbytes/s, TOTAL TIME= 0.0050( 2.0%)
STEPO_FPOS WAS : T00000000M
NSTEP = 0 STEPO 0AAA00AAA
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 309486308 BYTES IN 7 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0255( 1.2%)
04:40:48 STEP 0 H= 0:00 +CPU= 28.759
NSTEP = 1 STEPO 0AAA00AAA
04:41:10 STEP 1 H= 0:15 +CPU= 21.957
NSTEP = 2 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
04:39:57 0AA000000 STEPO 0 0.135 0.135 21.157 0:06 0:28 0.55515332309566E-07 2GB 0MB
04:39:58 FULLPOS-B DYNFPOS 0 1.122 1.122 1.128 0:08 0:29 0.55515332309566E-07 2GB 0MB
04:40:01 FULLPOS-V DYNFPOS 0 3.069 3.069 3.105 0:11 0:32 0.55515332309566E-07 2GB 0MB
04:40:02 FULLPOS-T DYNFPOS 0 0.606 0.606 0.613 0:11 0:32 0.55515332309566E-07 2GB 0MB
04:40:03 0AAA00AAA STEPO 0 0.565 0.565 0.571 0:12 0:33 0.55515332309566E-07 2GB 0MB
04:40:48 0AAA00AAA STEPO 1 23.058 23.058 45.346 0:35 1:18 0.55515332309566E-07 2GB 0MB
04:41:10 0AAA00AAA STEPO 2 21.957 21.957 22.060 0:57 1:40 0.55515332309566E-07 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /config/slots/5/progress_file_12287720.xml
<running_values>
<last_cpu_time>68.210000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>2</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
