climateprediction.net home page
Task 22264297

Task 22264297

Name oifs_43r3_ps_0745_1993050100_123_962_12179389_0
Workunit 12179389
Created 22 Dec 2022, 3:55:36 UTC
Sent 21 Jan 2023, 13:29:35 UTC
Report deadline 22 Mar 2023, 13:29:35 UTC
Received 22 Jan 2023, 16:32:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532321
Run time 22 hours 28 min 29 sec
CPU time 21 hours 42 min 51 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.24 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.12 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,775.63 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 337, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 76269
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21065
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21065:21065] [20230122:155418:1674399258:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:54:51 STEP  336 H= 336:00 +CPU= 31.253
  15:55:08 STEP  337 H= 337:00 +CPU= 16.849
  15:55:25 STEP  338 H= 338:00 +CPU= 17.161
  15:55:54 STEP  339 H= 339:00 +CPU= 28.374
  15:56:11 STEP  340 H= 340:00 +CPU= 17.263
  15:56:29 STEP  341 H= 341:00 +CPU= 17.328
  15:56:58 STEP  342 H= 342:00 +CPU= 28.560
  15:57:14 STEP  343 H= 343:00 +CPU= 16.733
  15:57:32 STEP  344 H= 344:00 +CPU= 17.091
  15:57:59 STEP  345 H= 345:00 +CPU= 27.470
  15:58:16 STEP  346 H= 346:00 +CPU= 17.020
  15:58:33 STEP  347 H= 347:00 +CPU= 16.740
  15:59:03 STEP  348 H= 348:00 +CPU= 30.076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000348
  15:59:20 STEP  349 H= 349:00 +CPU= 16.736
  15:59:37 STEP  350 H= 350:00 +CPU= 17.088
  16:00:04 STEP  351 H= 351:00 +CPU= 27.537
  16:00:22 STEP  352 H= 352:00 +CPU= 17.088
  16:00:38 STEP  353 H= 353:00 +CPU= 16.719
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 354, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 76647
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21127
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21127:21127] [20230122:160150:1674399710:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:02:24 STEP  336 H= 336:00 +CPU= 31.759
  16:02:41 STEP  337 H= 337:00 +CPU= 16.750
  16:02:58 STEP  338 H= 338:00 +CPU= 16.824
  16:03:25 STEP  339 H= 339:00 +CPU= 27.237
  16:03:42 STEP  340 H= 340:00 +CPU= 17.005
  16:04:00 STEP  341 H= 341:00 +CPU= 17.550
  16:04:28 STEP  342 H= 342:00 +CPU= 28.187
  16:04:45 STEP  343 H= 343:00 +CPU= 16.925
  16:05:03 STEP  344 H= 344:00 +CPU= 17.493
  16:05:31 STEP  345 H= 345:00 +CPU= 28.131
  16:05:48 STEP  346 H= 346:00 +CPU= 17.094
  16:06:05 STEP  347 H= 347:00 +CPU= 16.715
  16:06:35 STEP  348 H= 348:00 +CPU= 29.993
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000348
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 349, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 76935
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21172
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21172:21172] [20230122:160752:1674400072:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:08:25 STEP  336 H= 336:00 +CPU= 31.105
  16:08:43 STEP  337 H= 337:00 +CPU= 17.291
  16:09:00 STEP  338 H= 338:00 +CPU= 17.209
  16:09:28 STEP  339 H= 339:00 +CPU= 27.753
  16:09:45 STEP  340 H= 340:00 +CPU= 17.239
  16:10:02 STEP  341 H= 341:00 +CPU= 17.003
  16:10:31 STEP  342 H= 342:00 +CPU= 28.522
  16:10:48 STEP  343 H= 343:00 +CPU= 16.646
  16:11:05 STEP  344 H= 344:00 +CPU= 17.048
  16:11:32 STEP  345 H= 345:00 +CPU= 27.314
  16:11:49 STEP  346 H= 346:00 +CPU= 17.052
  16:12:06 STEP  347 H= 347:00 +CPU= 16.610
  16:12:36 STEP  348 H= 348:00 +CPU= 30.182
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000348
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 349, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 77224
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21217
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21217:21217] [20230122:161352:1674400432:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:14:25 STEP  336 H= 336:00 +CPU= 31.165
  16:14:42 STEP  337 H= 337:00 +CPU= 17.052
  16:14:59 STEP  338 H= 338:00 +CPU= 17.562
  16:15:27 STEP  339 H= 339:00 +CPU= 27.572
  16:15:45 STEP  340 H= 340:00 +CPU= 17.466
  16:16:02 STEP  341 H= 341:00 +CPU= 17.124
  16:16:30 STEP  342 H= 342:00 +CPU= 28.582
  16:16:47 STEP  343 H= 343:00 +CPU= 16.547
  16:17:04 STEP  344 H= 344:00 +CPU= 16.992
  16:17:32 STEP  345 H= 345:00 +CPU= 27.800
  16:17:49 STEP  346 H= 346:00 +CPU= 17.638
  16:18:06 STEP  347 H= 347:00 +CPU= 16.971
  16:18:37 STEP  348 H= 348:00 +CPU= 30.855
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000348
  16:18:55 STEP  349 H= 349:00 +CPU= 17.156
  16:19:12 STEP  350 H= 350:00 +CPU= 17.280
  16:19:39 STEP  351 H= 351:00 +CPU= 27.385
  16:19:57 STEP  352 H= 352:00 +CPU= 17.144
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 353, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 77593
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21308
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21308:21308] [20230122:162115:1674400875:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:21:48 STEP  336 H= 336:00 +CPU= 31.192
  16:22:05 STEP  337 H= 337:00 +CPU= 16.775
  16:22:21 STEP  338 H= 338:00 +CPU= 16.838
  16:22:49 STEP  339 H= 339:00 +CPU= 27.387
  16:23:06 STEP  340 H= 340:00 +CPU= 16.857
  16:23:23 STEP  341 H= 341:00 +CPU= 16.974
  16:23:51 STEP  342 H= 342:00 +CPU= 28.339
  16:24:08 STEP  343 H= 343:00 +CPU= 16.823
  16:24:26 STEP  344 H= 344:00 +CPU= 17.467
  16:24:54 STEP  345 H= 345:00 +CPU= 27.894
  16:25:11 STEP  346 H= 346:00 +CPU= 17.046
  16:25:28 STEP  347 H= 347:00 +CPU= 16.681
  16:25:58 STEP  348 H= 348:00 +CPU= 29.952
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000348
  16:26:15 STEP  349 H= 349:00 +CPU= 16.975
  16:26:32 STEP  350 H= 350:00 +CPU= 17.368
  16:27:00 STEP  351 H= 351:00 +CPU= 27.552
  16:27:18 STEP  352 H= 352:00 +CPU= 17.500
  16:27:35 STEP  353 H= 353:00 +CPU= 16.920
  16:28:03 STEP  354 H= 354:00 +CPU= 27.993
  16:28:19 STEP  355 H= 355:00 +CPU= 16.496
  16:28:36 STEP  356 H= 356:00 +CPU= 16.862
  16:29:03 STEP  357 H= 357:00 +CPU= 27.342
  16:29:20 STEP  358 H= 358:00 +CPU= 16.777
  16:29:37 STEP  359 H= 359:00 +CPU= 16.911
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0745
(argv4) batchid: 962
(argv5) wuid: 12179389
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5edfcf0d7d398c26ce2bb0e8ace699b4 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0745_1993050100_123_962_12179389.zip
ic_ancil_file: ic_ancil_12179389
ifsdata_file: ifsdata_12179389
climate_data_file: clim_data_12179389
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_0940f2fe6649f7683639ffbab6c8de58 to: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12179389.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1b1b1a5297d941fff2696d780260e1f1 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12179389.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12179389.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12179389.xml
-- Model is restarting --
Adjusting last_iter, 360, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12179389.xml
last_cpu_time: 78111
upload_file_number: 14
last_iter: 359
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561
The child process has been launched with process id: 21382
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:facebook-V:1:1:21382:21382] [20230122:163107:1674401467:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:31:41 STEP  360 H= 360:00 +CPU= 31.346
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179389/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0745_1993050100_123_962_12179389_0_r1011395561_14.zip
Uploading the intermediate file: upload_file_14.zip
  16:31:57 STEP  361 H= 361:00 +CPU= 16.573
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net