<core_client_version>7.16.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1i7
(argv4) batchid: 1018
(argv5) wuid: 12290278
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /home/mike/BOINC/slots/18
Project directory is: /home/mike/BOINC/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /home/mike/BOINC/projects/climateprediction.net/oifs_43r3_bl_12290278
Copying: /home/mike/BOINC/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/mike/BOINC/slots/18/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/mike/BOINC/slots/18/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2fab678c569ad00dd39cd251ccd5d638 to: /home/mike/BOINC/slots/18/oifs_43r3_bl_a1i7_2016092300_20_1018_12290278.zip
Unzipping the namelist zip file: /home/mike/BOINC/slots/18/oifs_43r3_bl_a1i7_2016092300_20_1018_12290278.zip
ic_ancil_file: ic_ancil_12290278
ifsdata_file: ifsdata_12290278
climate_data_file: clim_data_12290278
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/mike/BOINC/slots/18/ic_ancil_12290278.zip
Unzipping the IC ancils zip file: /home/mike/BOINC/slots/18/ic_ancil_12290278.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fecb94ee0d6c8322ec6ae84dca813e6f to: /home/mike/BOINC/slots/18/ifsdata/ifsdata_12290278.zip
Unzipping the ifsdata_zip file: /home/mike/BOINC/slots/18/ifsdata/ifsdata_12290278.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/mike/BOINC/slots/18/159l_2/clim_data_12290278.zip
Unzipping the climate data zip file: /home/mike/BOINC/slots/18/159l_2/clim_data_12290278.zip
Checking for progress XML file: /home/mike/BOINC/slots/18/progress_file_12290278.xml
Creating progress file: /home/mike/BOINC/slots/18/progress_file_12290278.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1i7_2016092300_20_1018_12290278_0_r724036174
The child process has been launched with process id: 101614
Executing the command: /home/mike/BOINC/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at 0x1)
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Test101:1:1:101614:101614] [20240613:075510:1718279710:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:55:49 STEP 0 H= 0:00 +CPU= 24.825
Moving to projects directory: /home/mike/BOINC/slots/18/ICMGGh7zg+000000
Moving to projects directory: /home/mike/BOINC/slots/18/ICMSHh7zg+000000
Moving to projects directory: /home/mike/BOINC/slots/18/ICMUAh7zg+000000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:56:21 STEP 1 H= 0:15 +CPU= 21.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:56:48 STEP 2 H= 0:30 +CPU= 20.978
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:57:17 STEP 3 H= 0:45 +CPU= 22.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SNOW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RAIN AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CLW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CIW AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM RFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM SFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CRFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM CSFL AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
GPNORM PFRC AVERAGE MINIMUM MAXIMUM
AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
NSTEP = 0 STEPO A00000000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=w
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 358599956 BYTES IN 6 RECORDS TRANSFERRED IN 0.0001 SECONDS******** Mbytes/s, TOTAL TIME= 2.0375( 0.0%)
NSTEP = 0 STEPO 0AA000000
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMSHh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMSHh7zg+000000
MPI-TASK: 1 - 707399250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0090 SECONDS 78599.9 Mbytes/s, TOTAL TIME= 0.1786( 5.0%)
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=w
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 4910292800 BYTES IN 125 RECORDS TRANSFERRED IN 0.0042 SECONDS******** Mbytes/s, TOTAL TIME= 0.1362( 3.1%)
STEPO_FPOS WAS : B00000F00E
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 10 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS 0.1 Mbytes/s, TOTAL TIME= 0.0042( 2.4%)
STEPO_FPOS WAS : V00000000Y
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
./ICMUAh7zg+000000 MODE=a
IO-STREAM CLOSED - ./ICMUAh7zg+000000
MPI-TASK: 1 - 298313571 BYTES IN 4 RECORDS TRANSFERRED IN 0.0003 SECONDS994378.6 Mbytes/s, TOTAL TIME= 0.0069( 4.3%)
STEPO_FPOS WAS : T00000000M
NSTEP = 0 STEPO 0AAA00AAA
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 0 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+000000 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+000000
MPI-TASK: 1 - 306893300 BYTES IN 7 RECORDS TRANSFERRED IN 0.0007 SECONDS438419.0 Mbytes/s, TOTAL TIME= 0.0162( 4.3%)
07:55:49 STEP 0 H= 0:00 +CPU= 24.825
NSTEP = 1 STEPO 0AAA00AAA
07:56:21 STEP 1 H= 0:15 +CPU= 21.046
NSTEP = 2 STEPO 0AAA00AAA
07:56:48 STEP 2 H= 0:30 +CPU= 20.978
NSTEP = 3 STEPO 0AAA00AAA
07:57:17 STEP 3 H= 0:45 +CPU= 22.918
NSTEP = 4 STEPO 0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
07:55:21 FULLPOS-V DYNFPOS 0 2.394 2.394 2.398 0:09 0:11 0.55515332309566E-07 2GB 0MB
07:55:22 FULLPOS-T DYNFPOS 0 0.467 0.467 0.468 0:09 0:11 0.55515332309566E-07 2GB 0MB
07:55:22 0AAA00AAA STEPO 0 0.467 0.467 0.468 0:10 0:12 0.55515332309566E-07 2GB 0MB
07:55:49 0AAA00AAA STEPO 1 20.336 20.336 26.451 0:30 0:38 0.55515332309566E-07 2GB 0MB
07:56:21 0AAA00AAA STEPO 2 21.046 21.046 31.999 0:51 1:10 0.55515332309566E-07 2GB 0MB
07:56:48 0AAA00AAA STEPO 3 20.978 20.978 27.077 1:12 1:37 0.55515332309566E-07 2GB 0MB
07:57:17 0AAA00AAA STEPO 4 22.919 22.919 29.033 1:35 2:06 0.55515332309566E-07 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /home/mike/BOINC/slots/18/progress_file_12290278.xml
<running_values>
<last_cpu_time>104.400000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>4</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
