climateprediction.net home page
Task 22280599

Task 22280599

Name oifs_43r3_ps_0169_2008050100_123_977_12193813_1
Workunit 12193813
Created 23 Dec 2022, 20:08:17 UTC
Sent 29 Dec 2022, 20:30:44 UTC
Report deadline 28 Jan 2023, 20:30:44 UTC
Received 1 Jan 2023, 3:25:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 23 hours 9 min 14 sec
CPU time 22 hours 32 min 22 sec
Validate state Invalid
Credit 4,354.56
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.15 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,535.41 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
23859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:56936:56936] [20221231:223859:1672522739:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:41:30 STEP 1008 H=1008:00 +CPU=130.746
  22:42:47 STEP 1009 H=1009:00 +CPU= 74.358
  22:44:04 STEP 1010 H=1010:00 +CPU= 73.885
  22:46:00 STEP 1011 H=1011:00 +CPU=112.024
  22:47:14 STEP 1012 H=1012:00 +CPU= 72.087
  22:48:30 STEP 1013 H=1013:00 +CPU= 72.821
  22:50:31 STEP 1014 H=1014:00 +CPU=117.481
  22:51:47 STEP 1015 H=1015:00 +CPU= 72.488
  22:53:05 STEP 1016 H=1016:00 +CPU= 74.561
  22:55:04 STEP 1017 H=1017:00 +CPU=115.087
  22:56:21 STEP 1018 H=1018:00 +CPU= 74.581
  22:57:33 STEP 1019 H=1019:00 +CPU= 69.747
  22:59:44 STEP 1020 H=1020:00 +CPU=125.940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001020
  23:00:59 STEP 1021 H=1021:00 +CPU= 72.484
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:29 STEP 1022 H=1022:00 +CPU= 76.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:04:38 STEP 1023 H=1023:00 +CPU=113.967
  23:05:57 STEP 1024 H=1024:00 +CPU= 75.991
  23:07:11 STEP 1025 H=1025:00 +CPU= 72.027
  23:09:08 STEP 1026 H=1026:00 +CPU=112.555
  23:10:22 STEP 1027 H=1027:00 +CPU= 71.298
  23:11:40 STEP 1028 H=1028:00 +CPU= 75.517
  23:13:39 STEP 1029 H=1029:00 +CPU=114.702
  23:14:56 STEP 1030 H=1030:00 +CPU= 74.517
  23:16:13 STEP 1031 H=1031:00 +CPU= 74.254
  23:18:32 STEP 1032 H=1032:00 +CPU=134.259
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526_42.zip
Uploading the intermediate file: upload_file_42.zip
  23:19:48 STEP 1033 H=1033:00 +CPU= 72.983
  23:20:57 STEP 1034 H=1034:00 +CPU= 67.127
  23:22:52 STEP 1035 H=1035:00 +CPU=111.183
  23:24:12 STEP 1036 H=1036:00 +CPU= 76.999
  23:25:27 STEP 1037 H=1037:00 +CPU= 71.960
  23:27:28 STEP 1038 H=1038:00 +CPU=116.694
  23:28:44 STEP 1039 H=1039:00 +CPU= 73.148
  23:30:02 STEP 1040 H=1040:00 +CPU= 75.141
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:32:07 STEP 1041 H=1041:00 +CPU=109.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:33:35 STEP 1042 H=1042:00 +CPU= 74.041
  23:34:49 STEP 1043 H=1043:00 +CPU= 72.064
  23:37:02 STEP 1044 H=1044:00 +CPU=127.476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044
  23:38:18 STEP 1045 H=1045:00 +CPU= 73.244
  23:39:36 STEP 1046 H=1046:00 +CPU= 74.581
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0169
(argv4) batchid: 977
(argv5) wuid: 12193813
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a64291b27939c37ed8121d35eb7dfe5b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
ic_ancil_file: ic_ancil_12193813
ifsdata_file: ifsdata_12193813
climate_data_file: clim_data_12193813
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_eb866ef7d3dc5dfb4cb90510d1e48dcd to: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3a82a7b6462a444681daa6dda5ff7eb1 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12193813.xml
-- Model is restarting --
Adjusting last_iter, 1047, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
last_cpu_time: 69189
upload_file_number: 43
last_iter: 1031
last_upload: 3715200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526
The child process has been launched with process id: 60425
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:60425:60425] [20221231:234116:1672526476:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:43:45 STEP 1032 H=1032:00 +CPU=129.310
  23:44:59 STEP 1033 H=1033:00 +CPU= 70.828
  23:46:18 STEP 1034 H=1034:00 +CPU= 75.928
  23:48:14 STEP 1035 H=1035:00 +CPU=111.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:50:56 STEP 1036 H=1036:00 +CPU= 75.467
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:40 STEP 1037 H=1037:00 +CPU= 57.066
  23:54:28 STEP 1038 H=1038:00 +CPU=103.996
  23:55:42 STEP 1039 H=1039:00 +CPU= 71.419
  23:56:56 STEP 1040 H=1040:00 +CPU= 71.689
  23:58:58 STEP 1041 H=1041:00 +CPU=117.426
  00:00:13 STEP 1042 H=1042:00 +CPU= 72.806
  00:01:29 STEP 1043 H=1043:00 +CPU= 73.198
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:03:56 STEP 1044 H=1044:00 +CPU=130.062
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001044
  00:05:11 STEP 1045 H=1045:00 +CPU= 72.247
  00:06:29 STEP 1046 H=1046:00 +CPU= 75.108
  00:08:27 STEP 1047 H=1047:00 +CPU=113.185
  00:09:45 STEP 1048 H=1048:00 +CPU= 75.228
  00:10:59 STEP 1049 H=1049:00 +CPU= 71.114
  00:13:02 STEP 1050 H=1050:00 +CPU=117.333
  00:14:17 STEP 1051 H=1051:00 +CPU= 72.396
  00:15:34 STEP 1052 H=1052:00 +CPU= 74.249
  00:17:31 STEP 1053 H=1053:00 +CPU=112.750
  00:18:48 STEP 1054 H=1054:00 +CPU= 74.227
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:21:05 STEP 1055 H=1055:00 +CPU= 72.799
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:24:21 STEP 1056 H=1056:00 +CPU=107.308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526_43.zip
Uploading the intermediate file: upload_file_43.zip
  00:25:27 STEP 1057 H=1057:00 +CPU= 63.595
  00:26:36 STEP 1058 H=1058:00 +CPU= 66.800
  00:28:20 STEP 1059 H=1059:00 +CPU=101.466
  00:29:39 STEP 1060 H=1060:00 +CPU= 76.196
  00:30:54 STEP 1061 H=1061:00 +CPU= 72.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:33:05 STEP 1062 H=1062:00 +CPU=115.257
  00:34:18 STEP 1063 H=1063:00 +CPU= 70.934
  00:35:33 STEP 1064 H=1064:00 +CPU= 73.044
  00:37:31 STEP 1065 H=1065:00 +CPU=113.460
  00:38:46 STEP 1066 H=1066:00 +CPU= 72.686
  00:40:03 STEP 1067 H=1067:00 +CPU= 73.408
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0169
(argv4) batchid: 977
(argv5) wuid: 12193813
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a64291b27939c37ed8121d35eb7dfe5b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
ic_ancil_file: ic_ancil_12193813
ifsdata_file: ifsdata_12193813
climate_data_file: clim_data_12193813
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_eb866ef7d3dc5dfb4cb90510d1e48dcd to: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3a82a7b6462a444681daa6dda5ff7eb1 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12193813.xml
-- Model is restarting --
Adjusting last_iter, 1068, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
last_cpu_time: 72399
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526
The child process has been launched with process id: 65045
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:65045:65045] [20230101:004310:1672530190:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:46:36 STEP 1056 H=1056:00 +CPU=102.078
  00:47:38 STEP 1057 H=1057:00 +CPU= 59.802
  00:48:47 STEP 1058 H=1058:00 +CPU= 67.094
  00:50:32 STEP 1059 H=1059:00 +CPU=101.660
  00:51:50 STEP 1060 H=1060:00 +CPU= 74.227
  00:53:04 STEP 1061 H=1061:00 +CPU= 71.748
  00:55:07 STEP 1062 H=1062:00 +CPU=117.026
  00:56:24 STEP 1063 H=1063:00 +CPU= 73.267
  00:57:39 STEP 1064 H=1064:00 +CPU= 72.385
  00:59:36 STEP 1065 H=1065:00 +CPU=112.480
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:01:39 STEP 1066 H=1066:00 +CPU= 72.903
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:04:56 STEP 1067 H=1067:00 +CPU= 68.009
  01:06:43 STEP 1068 H=1068:00 +CPU=103.071
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
  01:07:51 STEP 1069 H=1069:00 +CPU= 65.225
  01:08:59 STEP 1070 H=1070:00 +CPU= 64.990
  01:10:41 STEP 1071 H=1071:00 +CPU= 98.777
  01:11:47 STEP 1072 H=1072:00 +CPU= 64.559
  01:12:56 STEP 1073 H=1073:00 +CPU= 66.716
  01:14:48 STEP 1074 H=1074:00 +CPU=107.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:16:34 STEP 1075 H=1075:00 +CPU= 69.972
  01:17:47 STEP 1076 H=1076:00 +CPU= 69.867
  01:19:43 STEP 1077 H=1077:00 +CPU=111.442
  01:21:00 STEP 1078 H=1078:00 +CPU= 74.079
  01:22:12 STEP 1079 H=1079:00 +CPU= 69.984
  01:24:26 STEP 1080 H=1080:00 +CPU=129.304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526_44.zip
Uploading the intermediate file: upload_file_44.zip
  01:25:42 STEP 1081 H=1081:00 +CPU= 73.244
  01:26:59 STEP 1082 H=1082:00 +CPU= 74.329
  01:28:57 STEP 1083 H=1083:00 +CPU=114.421
  01:30:15 STEP 1084 H=1084:00 +CPU= 74.611
  01:31:28 STEP 1085 H=1085:00 +CPU= 71.145
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:33:49 STEP 1086 H=1086:00 +CPU=113.741
  01:35:03 STEP 1087 H=1087:00 +CPU= 71.517
  01:36:19 STEP 1088 H=1088:00 +CPU= 73.517
  01:38:19 STEP 1089 H=1089:00 +CPU=115.540
  01:39:35 STEP 1090 H=1090:00 +CPU= 72.934
  01:40:51 STEP 1091 H=1091:00 +CPU= 73.997
  01:43:02 STEP 1092 H=1092:00 +CPU=125.898
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0169
(argv4) batchid: 977
(argv5) wuid: 12193813
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a64291b27939c37ed8121d35eb7dfe5b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
ic_ancil_file: ic_ancil_12193813
ifsdata_file: ifsdata_12193813
climate_data_file: clim_data_12193813
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_eb866ef7d3dc5dfb4cb90510d1e48dcd to: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3a82a7b6462a444681daa6dda5ff7eb1 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12193813.xml
-- Model is restarting --
Adjusting last_iter, 1093, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
last_cpu_time: 75567
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526
The child process has been launched with process id: 69674
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:69674:69674] [20230101:014440:1672533880:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:47:04 STEP 1080 H=1080:00 +CPU=128.227
  01:48:19 STEP 1081 H=1081:00 +CPU= 72.961
  01:49:36 STEP 1082 H=1082:00 +CPU= 73.948
  01:51:34 STEP 1083 H=1083:00 +CPU=114.019
  01:52:52 STEP 1084 H=1084:00 +CPU= 75.487
  01:54:05 STEP 1085 H=1085:00 +CPU= 70.274
  01:56:05 STEP 1086 H=1086:00 +CPU=115.757
  01:57:19 STEP 1087 H=1087:00 +CPU= 70.778
  01:58:36 STEP 1088 H=1088:00 +CPU= 73.859
  02:00:37 STEP 1089 H=1089:00 +CPU=115.866
  02:01:53 STEP 1090 H=1090:00 +CPU= 73.896
  02:03:10 STEP 1091 H=1091:00 +CPU= 73.668
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:05:35 STEP 1092 H=1092:00 +CPU=128.340
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
  02:06:51 STEP 1093 H=1093:00 +CPU= 72.757
  02:08:06 STEP 1094 H=1094:00 +CPU= 72.752
  02:10:03 STEP 1095 H=1095:00 +CPU=112.883
  02:11:21 STEP 1096 H=1096:00 +CPU= 74.870
  02:12:36 STEP 1097 H=1097:00 +CPU= 72.040
  02:14:37 STEP 1098 H=1098:00 +CPU=116.847
  02:15:52 STEP 1099 H=1099:00 +CPU= 72.213
  02:17:09 STEP 1100 H=1100:00 +CPU= 74.097
  02:19:02 STEP 1101 H=1101:00 +CPU=109.121
  02:20:09 STEP 1102 H=1102:00 +CPU= 65.027
  02:21:24 STEP 1103 H=1103:00 +CPU= 72.128
  02:23:42 STEP 1104 H=1104:00 +CPU=133.262
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526_45.zip
Uploading the intermediate file: upload_file_45.zip
  02:24:57 STEP 1105 H=1105:00 +CPU= 72.022
  02:26:14 STEP 1106 H=1106:00 +CPU= 74.575
  02:28:09 STEP 1107 H=1107:00 +CPU=110.579
  02:29:29 STEP 1108 H=1108:00 +CPU= 76.935
  02:30:45 STEP 1109 H=1109:00 +CPU= 72.915
  02:32:44 STEP 1110 H=1110:00 +CPU=115.158
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:34:10 STEP 1111 H=1111:00 +CPU= 70.963
  02:35:29 STEP 1112 H=1112:00 +CPU= 76.280
  02:37:30 STEP 1113 H=1113:00 +CPU=116.550
  02:38:47 STEP 1114 H=1114:00 +CPU= 74.296
  02:40:03 STEP 1115 H=1115:00 +CPU= 72.945
  02:42:19 STEP 1116 H=1116:00 +CPU=129.865
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116
  02:43:36 STEP 1117 H=1117:00 +CPU= 72.838
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0169
(argv4) batchid: 977
(argv5) wuid: 12193813
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193813
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a64291b27939c37ed8121d35eb7dfe5b to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0169_2008050100_123_977_12193813.zip
ic_ancil_file: ic_ancil_12193813
ifsdata_file: ifsdata_12193813
climate_data_file: clim_data_12193813
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_eb866ef7d3dc5dfb4cb90510d1e48dcd to: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12193813.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3a82a7b6462a444681daa6dda5ff7eb1 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12193813.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12193813.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12193813.xml
-- Model is restarting --
Adjusting last_iter, 1118, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12193813.xml
last_cpu_time: 78996
upload_file_number: 46
last_iter: 1103
last_upload: 3974400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0169_2008050100_123_977_12193813_1_r1100675526
The child process has been launched with process id: 74420
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:74420:74420] [20230101:024700:1672537620:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:49:17 STEP 1104 H=1104:00 +CPU=119.637
  02:50:21 STEP 1105 H=1105:00 +CPU= 62.249
  02:51:31 STEP 1106 H=1106:00 +CPU= 67.723
  02:53:26 STEP 1107 H=1107:00 +CPU=110.736
  02:54:42 STEP 1108 H=1108:00 +CPU= 73.034
  02:55:59 STEP 1109 H=1109:00 +CPU= 73.858
  02:57:56 STEP 1110 H=1110:00 +CPU=112.419
  02:59:11 STEP 1111 H=1111:00 +CPU= 72.228
  03:00:29 STEP 1112 H=1112:00 +CPU= 75.224
  03:02:27 STEP 1113 H=1113:00 +CPU=113.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:04:09 STEP 1114 H=1114:00 +CPU= 76.613
  03:05:25 STEP 1115 H=1115:00 +CPU= 72.783
  03:07:37 STEP 1116 H=1116:00 +CPU=127.636
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116
  03:08:51 STEP 1117 H=1117:00 +CPU= 70.842
  03:10:08 STEP 1118 H=1118:00 +CPU= 74.498
  03:12:10 STEP 1119 H=1119:00 +CPU=116.727
  03:13:27 STEP 1120 H=1120:00 +CPU= 73.696
  03:14:44 STEP 1121 H=1121:00 +CPU= 73.831
  03:16:45 STEP 1122 H=1122:00 +CPU=116.310
  03:18:01 STEP 1123 H=1123:00 +CPU= 73.220
  03:19:19 STEP 1124 H=1124:00 +CPU= 74.995
  03:21:17 STEP 1125 H=1125:00 +CPU=112.790
  03:22:34 STEP 1126 H=1126:00 +CPU= 74.091
  03:23:53 STEP 1127 H=1127:00 +CPU= 75.634
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:03:40 1467890 22280599 oifs_43r3_ps_0169_2008050100_123_977_12193813_1 3,110,400 54,882 0.0176
09 May 2023 16:03:29 1467890 22280599 oifs_43r3_ps_0169_2008050100_123_977_12193813_1 2,246,400 40,999 0.0183
09 May 2023 16:02:27 1467890 22280599 oifs_43r3_ps_0169_2008050100_123_977_12193813_1 864,000 21,629 0.0250


©2024 climateprediction.net