<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a4hh
(argv4) batchid: 1018
(argv5) wuid: 12294140
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /config/slots/0
Project directory is: /config/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /config/projects/climateprediction.net/oifs_43r3_bl_12294140
Copying: /config/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /config/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /config/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f6efd867a5ff376a983cbb7bf4e34eed to: /config/slots/0/oifs_43r3_bl_a4hh_2016092300_20_1018_12294140.zip
Unzipping the namelist zip file: /config/slots/0/oifs_43r3_bl_a4hh_2016092300_20_1018_12294140.zip
ic_ancil_file: ic_ancil_12294140
ifsdata_file: ifsdata_12294140
climate_data_file: clim_data_12294140
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /config/slots/0/ic_ancil_12294140.zip
Unzipping the IC ancils zip file: /config/slots/0/ic_ancil_12294140.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_64530bb0abe77523056a4e29c2c1803b to: /config/slots/0/ifsdata/ifsdata_12294140.zip
Unzipping the ifsdata_zip file: /config/slots/0/ifsdata/ifsdata_12294140.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /config/slots/0/159l_2/clim_data_12294140.zip
Unzipping the climate data zip file: /config/slots/0/159l_2/clim_data_12294140.zip
Checking for progress XML file: /config/slots/0/progress_file_12294140.xml
Creating progress file: /config/slots/0/progress_file_12294140.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a4hh_2016092300_20_1018_12294140_0_r1984510117
The child process has been launched with process id: 5997
Executing the command: /config/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:995674c7b970:1:1:5997:5997] [20240614:091950:1718353190:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc
UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED
SUECAEC: INLEV,ISYEAR,IEYEAR= 0 1957 2010
suecaec NAERMACC = 1
UPDT RAESHDU = 2000.0
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20160923
90 9.999999999999998E-005
DATE= 2016 9 23
NORMS AT NSTEP CNT4 0
SPECTRAL NORMS - LOG(PREHYDS) 0.115260884514965E+02
LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY
AVE 0.483197135618192E-05 0.555153323095658E-07 0.201280633805116E+03 0.896372355588448E+02
GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM
AVE 0.116648129292729E-02 0.000000000000000E+00 0.108251512983811E-01
GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
NSTEP = 0 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 369517236 BYTES IN 6 RECORDS TRANSFERRED IN 0.0001 SECONDS******** Mbytes/s, TOTAL TIME= 0.0412( 0.2%)
NSTEP = 0 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000000
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0287 SECONDS 24648.1 Mbytes/s, TOTAL TIME= 0.2633(10.9%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 5084969280 BYTES IN 125 RECORDS TRANSFERRED IN 0.0030 SECONDS******** Mbytes/s, TOTAL TIME= 0.1173( 2.6%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0028( 3.6%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 309230851 BYTES IN 4 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0047( 4.3%)
STEPO_FPOS WAS : T00000000M
NSTEP = 0 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 7 lines from file: ifs.stat
09:19:51 000000000 CNT3 -999 0.465 0.465 0.475 0:00 0:00 0.00000000000000E+00 1GB 0MB
09:19:58 A00000000 STEPO 0 4.210 4.210 8.256 0:04 0:08 0.55515332309566E-07 2GB 0MB
09:19:59 0AA000000 STEPO 0 0.049 0.049 0.049 0:04 0:08 0.55515332309566E-07 2GB 0MB
09:20:00 FULLPOS-B DYNFPOS 0 0.973 0.973 0.999 0:05 0:09 0.55515332309566E-07 2GB 0MB
09:20:04 FULLPOS-V DYNFPOS 0 2.458 2.458 4.530 0:07 0:13 0.55515332309566E-07 2GB 0MB
09:20:06 FULLPOS-T DYNFPOS 0 0.348 0.348 2.351 0:08 0:16 0.55515332309566E-07 2GB 0MB
09:20:07 0AAA00AAA STEPO 0 0.309 0.309 0.310 0:08 0:16 0.55515332309566E-07 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /config/slots/0/progress_file_12294140.xml
<running_values>
<last_cpu_time>8.850000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>0</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>