climateprediction.net home page
Task 22252893

Task 22252893

Name oifs_43r3_ps_0984_1982050100_123_951_12168628_0
Workunit 12168628
Created 20 Dec 2022, 23:34:02 UTC
Sent 20 Dec 2022, 23:37:37 UTC
Report deadline 19 Jan 2023, 23:37:37 UTC
Received 22 Dec 2022, 6:30:06 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1467890
Run time 13 hours 27 min 24 sec
CPU time 12 hours 55 min 20 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.68 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,680.31 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
8:26 STEP  962 H= 962:00 +CPU= 30.779
  02:39:18 STEP  963 H= 963:00 +CPU= 51.056
  02:39:49 STEP  964 H= 964:00 +CPU= 31.252
  02:40:21 STEP  965 H= 965:00 +CPU= 32.239
  02:41:12 STEP  966 H= 966:00 +CPU= 50.763
  02:41:45 STEP  967 H= 967:00 +CPU= 32.251
  02:42:17 STEP  968 H= 968:00 +CPU= 32.306
  02:43:08 STEP  969 H= 969:00 +CPU= 50.962
  02:43:41 STEP  970 H= 970:00 +CPU= 32.474
  02:44:11 STEP  971 H= 971:00 +CPU= 30.104
  02:45:08 STEP  972 H= 972:00 +CPU= 57.339
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000972
  02:45:39 STEP  973 H= 973:00 +CPU= 30.916
  02:46:14 STEP  974 H= 974:00 +CPU= 34.330
  02:47:03 STEP  975 H= 975:00 +CPU= 48.876
  02:47:37 STEP  976 H= 976:00 +CPU= 33.944
  02:48:07 STEP  977 H= 977:00 +CPU= 29.777
  02:48:59 STEP  978 H= 978:00 +CPU= 52.457
  02:49:29 STEP  979 H= 979:00 +CPU= 29.664
  02:50:03 STEP  980 H= 980:00 +CPU= 33.846
  02:50:54 STEP  981 H= 981:00 +CPU= 50.545
  02:51:27 STEP  982 H= 982:00 +CPU= 33.186
  02:51:59 STEP  983 H= 983:00 +CPU= 31.507
  02:52:57 STEP  984 H= 984:00 +CPU= 58.352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_40.zip
Uploading the intermediate file: upload_file_40.zip
  02:53:29 STEP  985 H= 985:00 +CPU= 31.465
  02:54:00 STEP  986 H= 986:00 +CPU= 31.350
  02:54:52 STEP  987 H= 987:00 +CPU= 51.137
  02:55:24 STEP  988 H= 988:00 +CPU= 32.198
  02:55:57 STEP  989 H= 989:00 +CPU= 33.096
  02:56:48 STEP  990 H= 990:00 +CPU= 50.756
  02:57:21 STEP  991 H= 991:00 +CPU= 32.784
  02:57:52 STEP  992 H= 992:00 +CPU= 31.042
  02:58:44 STEP  993 H= 993:00 +CPU= 51.795
  02:59:15 STEP  994 H= 994:00 +CPU= 30.908
  02:59:48 STEP  995 H= 995:00 +CPU= 32.798
  03:00:43 STEP  996 H= 996:00 +CPU= 55.076
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000996
  03:01:17 STEP  997 H= 997:00 +CPU= 33.165
  03:01:49 STEP  998 H= 998:00 +CPU= 31.787
  03:02:41 STEP  999 H= 999:00 +CPU= 52.354
  03:03:12 STEP 1000 H=1000:00 +CPU= 30.959
  03:03:43 STEP 1001 H=1001:00 +CPU= 30.548
  03:04:37 STEP 1002 H=1002:00 +CPU= 53.851
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0984
(argv4) batchid: 951
(argv5) wuid: 12168628
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3718b7bf8542a357d65a792d40194645 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
ic_ancil_file: ic_ancil_12168628
ifsdata_file: ifsdata_12168628
climate_data_file: clim_data_12168628
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f4e78571da8a0c5453f34fe1f6b4e896 to: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f1482d78a027482f97b069c6e7dd0d2e to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168628.xml
-- Model is restarting --
Adjusting last_iter, 1003, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
last_cpu_time: 37220
upload_file_number: 41
last_iter: 983
last_upload: 3542400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270
The child process has been launched with process id: 894232
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:894232:894232] [20221222:030553:1671674753:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:07:30 STEP  984 H= 984:00 +CPU= 77.099
  03:08:16 STEP  985 H= 985:00 +CPU= 44.863
  03:09:02 STEP  986 H= 986:00 +CPU= 45.969
  03:10:15 STEP  987 H= 987:00 +CPU= 71.627
  03:10:58 STEP  988 H= 988:00 +CPU= 42.756
  03:11:26 STEP  989 H= 989:00 +CPU= 28.536
  03:12:17 STEP  990 H= 990:00 +CPU= 50.853
  03:12:49 STEP  991 H= 991:00 +CPU= 31.583
  03:13:21 STEP  992 H= 992:00 +CPU= 32.154
  03:14:11 STEP  993 H= 993:00 +CPU= 49.287
  03:14:53 STEP  994 H= 994:00 +CPU= 41.819
  03:15:18 STEP  995 H= 995:00 +CPU= 24.438
  03:16:17 STEP  996 H= 996:00 +CPU= 57.015
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+000996
  03:16:48 STEP  997 H= 997:00 +CPU= 31.079
  03:17:22 STEP  998 H= 998:00 +CPU= 33.377
  03:18:13 STEP  999 H= 999:00 +CPU= 51.687
  03:19:00 STEP 1000 H=1000:00 +CPU= 46.221
  03:19:45 STEP 1001 H=1001:00 +CPU= 44.993
  03:20:39 STEP 1002 H=1002:00 +CPU= 53.710
  03:21:14 STEP 1003 H=1003:00 +CPU= 32.609
  03:21:45 STEP 1004 H=1004:00 +CPU= 31.023
  03:22:42 STEP 1005 H=1005:00 +CPU= 57.206
  03:23:28 STEP 1006 H=1006:00 +CPU= 45.422
  03:24:15 STEP 1007 H=1007:00 +CPU= 46.122
  03:25:11 STEP 1008 H=1008:00 +CPU= 54.231
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_41.zip
Uploading the intermediate file: upload_file_41.zip
  03:25:43 STEP 1009 H=1009:00 +CPU= 31.604
  03:26:17 STEP 1010 H=1010:00 +CPU= 33.593
  03:27:08 STEP 1011 H=1011:00 +CPU= 50.188
  03:27:41 STEP 1012 H=1012:00 +CPU= 33.818
  03:28:23 STEP 1013 H=1013:00 +CPU= 41.179
  03:29:38 STEP 1014 H=1014:00 +CPU= 73.744
  03:30:11 STEP 1015 H=1015:00 +CPU= 32.116
  03:30:44 STEP 1016 H=1016:00 +CPU= 33.457
  03:31:35 STEP 1017 H=1017:00 +CPU= 50.431
  03:32:08 STEP 1018 H=1018:00 +CPU= 33.567
  03:32:41 STEP 1019 H=1019:00 +CPU= 32.366
  03:33:41 STEP 1020 H=1020:00 +CPU= 57.988
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001020
  03:34:12 STEP 1021 H=1021:00 +CPU= 31.493
  03:34:46 STEP 1022 H=1022:00 +CPU= 33.589
  03:35:38 STEP 1023 H=1023:00 +CPU= 51.738
  03:36:11 STEP 1024 H=1024:00 +CPU= 33.267
  03:36:45 STEP 1025 H=1025:00 +CPU= 33.373
  03:37:37 STEP 1026 H=1026:00 +CPU= 51.792
  03:38:21 STEP 1027 H=1027:00 +CPU= 44.156
  03:39:08 STEP 1028 H=1028:00 +CPU= 46.555
  03:40:05 STEP 1029 H=1029:00 +CPU= 56.295
  03:40:40 STEP 1030 H=1030:00 +CPU= 34.859
  03:41:14 STEP 1031 H=1031:00 +CPU= 32.152
  03:42:22 STEP 1032 H=1032:00 +CPU= 61.927
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_42.zip
Uploading the intermediate file: upload_file_42.zip
  03:42:54 STEP 1033 H=1033:00 +CPU= 32.091
  03:43:26 STEP 1034 H=1034:00 +CPU= 32.156
  03:44:18 STEP 1035 H=1035:00 +CPU= 52.029
  03:44:50 STEP 1036 H=1036:00 +CPU= 31.420
  03:45:23 STEP 1037 H=1037:00 +CPU= 33.171
  03:46:16 STEP 1038 H=1038:00 +CPU= 52.634
  03:46:50 STEP 1039 H=1039:00 +CPU= 33.948
  03:47:24 STEP 1040 H=1040:00 +CPU= 33.726
  03:48:16 STEP 1041 H=1041:00 +CPU= 52.307
  03:48:50 STEP 1042 H=1042:00 +CPU= 33.254
  03:49:22 STEP 1043 H=1043:00 +CPU= 32.202
  03:50:21 STEP 1044 H=1044:00 +CPU= 58.764
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001044
  03:50:54 STEP 1045 H=1045:00 +CPU= 33.080
  03:51:28 STEP 1046 H=1046:00 +CPU= 33.982
  03:52:21 STEP 1047 H=1047:00 +CPU= 52.619
  03:52:55 STEP 1048 H=1048:00 +CPU= 33.711
  03:53:28 STEP 1049 H=1049:00 +CPU= 32.730
  03:54:22 STEP 1050 H=1050:00 +CPU= 53.642
  03:54:54 STEP 1051 H=1051:00 +CPU= 32.468
  03:55:29 STEP 1052 H=1052:00 +CPU= 35.035
  03:56:20 STEP 1053 H=1053:00 +CPU= 50.299
  03:56:55 STEP 1054 H=1054:00 +CPU= 35.318
  03:57:29 STEP 1055 H=1055:00 +CPU= 33.595
  03:58:41 STEP 1056 H=1056:00 +CPU= 71.161
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_43.zip
Uploading the intermediate file: upload_file_43.zip
  03:59:26 STEP 1057 H=1057:00 +CPU= 45.014
  04:00:15 STEP 1058 H=1058:00 +CPU= 47.453
  04:01:28 STEP 1059 H=1059:00 +CPU= 72.256
  04:02:02 STEP 1060 H=1060:00 +CPU= 34.239
  04:02:36 STEP 1061 H=1061:00 +CPU= 33.217
  04:03:27 STEP 1062 H=1062:00 +CPU= 50.520
  04:04:05 STEP 1063 H=1063:00 +CPU= 38.467
  04:04:51 STEP 1064 H=1064:00 +CPU= 45.748
  04:05:48 STEP 1065 H=1065:00 +CPU= 56.570
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0984
(argv4) batchid: 951
(argv5) wuid: 12168628
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3718b7bf8542a357d65a792d40194645 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
ic_ancil_file: ic_ancil_12168628
ifsdata_file: ifsdata_12168628
climate_data_file: clim_data_12168628
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f4e78571da8a0c5453f34fe1f6b4e896 to: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f1482d78a027482f97b069c6e7dd0d2e to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168628.xml
-- Model is restarting --
Adjusting last_iter, 1066, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
last_cpu_time: 40769
upload_file_number: 44
last_iter: 1055
last_upload: 3801600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270
The child process has been launched with process id: 897803
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:897803:897803] [20221222:040739:1671678459:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:09:22 STEP 1056 H=1056:00 +CPU= 62.453
  04:10:07 STEP 1057 H=1057:00 +CPU= 44.879
  04:10:54 STEP 1058 H=1058:00 +CPU= 46.880
  04:12:08 STEP 1059 H=1059:00 +CPU= 73.055
  04:12:54 STEP 1060 H=1060:00 +CPU= 45.948
  04:13:25 STEP 1061 H=1061:00 +CPU= 30.453
  04:14:17 STEP 1062 H=1062:00 +CPU= 52.580
  04:15:00 STEP 1063 H=1063:00 +CPU= 39.827
  04:15:32 STEP 1064 H=1064:00 +CPU= 31.848
  04:16:42 STEP 1065 H=1065:00 +CPU= 69.517
  04:17:30 STEP 1066 H=1066:00 +CPU= 47.780
  04:18:15 STEP 1067 H=1067:00 +CPU= 44.416
  04:19:25 STEP 1068 H=1068:00 +CPU= 68.189
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001068
  04:19:57 STEP 1069 H=1069:00 +CPU= 32.354
  04:20:31 STEP 1070 H=1070:00 +CPU= 32.825
  04:21:40 STEP 1071 H=1071:00 +CPU= 68.427
  04:22:30 STEP 1072 H=1072:00 +CPU= 48.852
  04:23:00 STEP 1073 H=1073:00 +CPU= 30.028
  04:23:51 STEP 1074 H=1074:00 +CPU= 51.068
  04:24:23 STEP 1075 H=1075:00 +CPU= 31.353
  04:24:56 STEP 1076 H=1076:00 +CPU= 33.703
  04:25:50 STEP 1077 H=1077:00 +CPU= 53.104
  04:26:24 STEP 1078 H=1078:00 +CPU= 33.563
  04:27:02 STEP 1079 H=1079:00 +CPU= 34.566
  04:28:11 STEP 1080 H=1080:00 +CPU= 61.732
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_44.zip
Uploading the intermediate file: upload_file_44.zip
  04:28:49 STEP 1081 H=1081:00 +CPU= 38.690
  04:29:36 STEP 1082 H=1082:00 +CPU= 45.963
  04:30:49 STEP 1083 H=1083:00 +CPU= 72.697
  04:31:37 STEP 1084 H=1084:00 +CPU= 47.362
  04:32:23 STEP 1085 H=1085:00 +CPU= 45.100
  04:33:39 STEP 1086 H=1086:00 +CPU= 74.191
  04:34:25 STEP 1087 H=1087:00 +CPU= 46.255
  04:35:06 STEP 1088 H=1088:00 +CPU= 40.817
  04:35:56 STEP 1089 H=1089:00 +CPU= 49.618
  04:36:30 STEP 1090 H=1090:00 +CPU= 33.033
  04:37:02 STEP 1091 H=1091:00 +CPU= 32.382
  04:38:02 STEP 1092 H=1092:00 +CPU= 58.692
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001092
  04:38:35 STEP 1093 H=1093:00 +CPU= 32.853
  04:39:09 STEP 1094 H=1094:00 +CPU= 33.763
  04:40:02 STEP 1095 H=1095:00 +CPU= 52.588
  04:40:35 STEP 1096 H=1096:00 +CPU= 33.551
  04:41:10 STEP 1097 H=1097:00 +CPU= 33.539
  04:42:05 STEP 1098 H=1098:00 +CPU= 54.722
  04:42:38 STEP 1099 H=1099:00 +CPU= 32.551
  04:43:12 STEP 1100 H=1100:00 +CPU= 34.324
  04:44:23 STEP 1101 H=1101:00 +CPU= 70.161
  04:45:10 STEP 1102 H=1102:00 +CPU= 46.864
  04:46:00 STEP 1103 H=1103:00 +CPU= 46.070
  04:47:11 STEP 1104 H=1104:00 +CPU= 66.921
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_45.zip
Uploading the intermediate file: upload_file_45.zip
  04:47:45 STEP 1105 H=1105:00 +CPU= 33.671
  04:48:18 STEP 1106 H=1106:00 +CPU= 32.669
  04:49:12 STEP 1107 H=1107:00 +CPU= 53.917
  04:49:53 STEP 1108 H=1108:00 +CPU= 39.525
  04:50:38 STEP 1109 H=1109:00 +CPU= 44.984
  04:51:53 STEP 1110 H=1110:00 +CPU= 74.523
  04:52:39 STEP 1111 H=1111:00 +CPU= 45.692
  04:53:26 STEP 1112 H=1112:00 +CPU= 46.689
  04:54:36 STEP 1113 H=1113:00 +CPU= 69.740
  04:55:08 STEP 1114 H=1114:00 +CPU= 31.607
  04:55:42 STEP 1115 H=1115:00 +CPU= 33.736
  04:56:39 STEP 1116 H=1116:00 +CPU= 56.604
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001116
  04:57:14 STEP 1117 H=1117:00 +CPU= 34.570
  04:57:48 STEP 1118 H=1118:00 +CPU= 33.277
  04:58:40 STEP 1119 H=1119:00 +CPU= 52.460
  04:59:15 STEP 1120 H=1120:00 +CPU= 34.176
  04:59:49 STEP 1121 H=1121:00 +CPU= 33.289
  05:01:04 STEP 1122 H=1122:00 +CPU= 74.562
  05:01:50 STEP 1123 H=1123:00 +CPU= 45.357
  05:02:36 STEP 1124 H=1124:00 +CPU= 46.340
  05:03:49 STEP 1125 H=1125:00 +CPU= 72.179
  05:04:25 STEP 1126 H=1126:00 +CPU= 36.125
  05:05:02 STEP 1127 H=1127:00 +CPU= 32.842
  05:06:14 STEP 1128 H=1128:00 +CPU= 59.969
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_46.zip
Uploading the intermediate file: upload_file_46.zip
  05:06:51 STEP 1129 H=1129:00 +CPU= 35.209
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0984
(argv4) batchid: 951
(argv5) wuid: 12168628
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3718b7bf8542a357d65a792d40194645 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
ic_ancil_file: ic_ancil_12168628
ifsdata_file: ifsdata_12168628
climate_data_file: clim_data_12168628
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f4e78571da8a0c5453f34fe1f6b4e896 to: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f1482d78a027482f97b069c6e7dd0d2e to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168628.xml
-- Model is restarting --
Adjusting last_iter, 1130, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
last_cpu_time: 44234
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270
The child process has been launched with process id: 901985
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:901985:901985] [20221222:050856:1671682136:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:11:28 STEP 1128 H=1128:00 +CPU= 84.553
  05:12:13 STEP 1129 H=1129:00 +CPU= 44.807
  05:12:46 STEP 1130 H=1130:00 +CPU= 32.363
  05:13:39 STEP 1131 H=1131:00 +CPU= 52.427
  05:14:11 STEP 1132 H=1132:00 +CPU= 32.272
  05:14:45 STEP 1133 H=1133:00 +CPU= 33.542
  05:15:37 STEP 1134 H=1134:00 +CPU= 52.217
  05:16:12 STEP 1135 H=1135:00 +CPU= 34.609
  05:16:45 STEP 1136 H=1136:00 +CPU= 32.852
  05:17:37 STEP 1137 H=1137:00 +CPU= 51.684
  05:18:12 STEP 1138 H=1138:00 +CPU= 33.794
  05:18:44 STEP 1139 H=1139:00 +CPU= 32.643
  05:19:50 STEP 1140 H=1140:00 +CPU= 58.087
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001140
  05:20:26 STEP 1141 H=1141:00 +CPU= 35.353
  05:21:11 STEP 1142 H=1142:00 +CPU= 45.646
  05:22:24 STEP 1143 H=1143:00 +CPU= 72.465
  05:23:11 STEP 1144 H=1144:00 +CPU= 46.587
  05:23:51 STEP 1145 H=1145:00 +CPU= 39.760
  05:24:37 STEP 1146 H=1146:00 +CPU= 45.385
  05:25:22 STEP 1147 H=1147:00 +CPU= 45.012
  05:26:08 STEP 1148 H=1148:00 +CPU= 46.168
  05:27:13 STEP 1149 H=1149:00 +CPU= 64.205
  05:27:40 STEP 1150 H=1150:00 +CPU= 26.541
  05:28:14 STEP 1151 H=1151:00 +CPU= 26.434
  05:29:28 STEP 1152 H=1152:00 +CPU= 59.364
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270_47.zip
Uploading the intermediate file: upload_file_47.zip
  05:30:02 STEP 1153 H=1153:00 +CPU= 33.662
  05:30:35 STEP 1154 H=1154:00 +CPU= 32.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:35:19 STEP 1155 H=1155:00 +CPU= 53.248
  05:35:56 STEP 1156 H=1156:00 +CPU= 30.737
  05:36:26 STEP 1157 H=1157:00 +CPU= 29.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0984
(argv4) batchid: 951
(argv5) wuid: 12168628
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168628
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3718b7bf8542a357d65a792d40194645 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0984_1982050100_123_951_12168628.zip
ic_ancil_file: ic_ancil_12168628
ifsdata_file: ifsdata_12168628
climate_data_file: clim_data_12168628
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f4e78571da8a0c5453f34fe1f6b4e896 to: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12168628.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f1482d78a027482f97b069c6e7dd0d2e to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12168628.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12168628.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12168628.xml
-- Model is restarting --
Adjusting last_iter, 1158, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12168628.xml
last_cpu_time: 45604
upload_file_number: 48
last_iter: 1151
last_upload: 4147200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0984_1982050100_123_951_12168628_0_r1951246270
The child process has been launched with process id: 905827
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:905827:905827] [20221222:062337:1671686617:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:26:03 STEP 1152 H=1152:00 +CPU= 49.450
  06:26:33 STEP 1153 H=1153:00 +CPU= 29.183
  06:27:03 STEP 1154 H=1154:00 +CPU= 30.398
  06:27:43 STEP 1155 H=1155:00 +CPU= 39.586
  06:28:14 STEP 1156 H=1156:00 +CPU= 31.043
  06:28:46 STEP 1157 H=1157:00 +CPU= 31.085
  06:29:31 STEP 1158 H=1158:00 +CPU= 45.489
  06:30:04 STEP 1159 H=1159:00 +CPU= 31.617
  06:30:37 STEP 1160 H=1160:00 +CPU= 33.107
  06:31:28 STEP 1161 H=1161:00 +CPU= 50.950
  06:32:01 STEP 1162 H=1162:00 +CPU= 32.272
  06:32:33 STEP 1163 H=1163:00 +CPU= 32.654
  06:33:29 STEP 1164 H=1164:00 +CPU= 54.828
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+001164
  06:34:02 STEP 1165 H=1165:00 +CPU= 32.559
  06:34:34 STEP 1166 H=1166:00 +CPU= 32.325
  06:35:27 STEP 1167 H=1167:00 +CPU= 52.947
  06:36:00 STEP 1168 H=1168:00 +CPU= 33.012
  06:36:33 STEP 1169 H=1169:00 +CPU= 32.623
  06:37:25 STEP 1170 H=1170:00 +CPU= 51.180
  06:38:00 STEP 1171 H=1171:00 +CPU= 33.152
  06:38:31 STEP 1172 H=1172:00 +CPU= 31.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:39:55 STEP 1173 H=1173:00 +CPU= 50.565
  06:40:29 STEP 1174 H=1174:00 +CPU= 33.533
  06:41:01 STEP 1175 H=1175:00 +CPU= 31.932
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:45:20 1467890 22252893 oifs_43r3_ps_0984_1982050100_123_951_12168628_0 1,728,000 15,084 0.0087
09 May 2023 15:45:15 1467890 22252893 oifs_43r3_ps_0984_1982050100_123_951_12168628_0 864,000 7,389 0.0086


©2024 climateprediction.net