climateprediction.net home page
Task 22262383

Task 22262383

Name oifs_43r3_ps_0940_1991050100_123_960_12177584_0
Workunit 12177584
Created 21 Dec 2022, 23:07:08 UTC
Sent 4 Jan 2023, 22:20:34 UTC
Report deadline 3 Feb 2023, 22:20:34 UTC
Received 8 Jan 2023, 2:53:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1536968
Run time 1 days 4 hours 59 min 42 sec
CPU time 12 hours 25 min 56 sec
Validate state Invalid
Credit 11,491.20
Device peak FLOPS 8.32 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.72 MB
Peak swap size 4,980.85 MB
Peak disk usage 7,553.40 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
004
  05:18:45 STEP 2005 H=2005:00 +CPU= 14.279
  05:19:10 STEP 2006 H=2006:00 +CPU= 14.589
  05:19:43 STEP 2007 H=2007:00 +CPU= 22.370
  05:20:07 STEP 2008 H=2008:00 +CPU= 14.814
  05:20:37 STEP 2009 H=2009:00 +CPU= 14.464
  05:21:12 STEP 2010 H=2010:00 +CPU= 22.982
  05:21:38 STEP 2011 H=2011:00 +CPU= 14.103
  05:22:09 STEP 2012 H=2012:00 +CPU= 14.939
  05:22:48 STEP 2013 H=2013:00 +CPU= 22.383
  05:23:15 STEP 2014 H=2014:00 +CPU= 14.551
  05:23:37 STEP 2015 H=2015:00 +CPU= 14.525
  05:24:33 STEP 2016 H=2016:00 +CPU= 27.077
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_83.zip
Uploading the intermediate file: upload_file_83.zip
  05:24:58 STEP 2017 H=2017:00 +CPU= 14.527
  05:25:28 STEP 2018 H=2018:00 +CPU= 15.189
  05:26:13 STEP 2019 H=2019:00 +CPU= 23.139
  05:26:39 STEP 2020 H=2020:00 +CPU= 14.726
  05:27:15 STEP 2021 H=2021:00 +CPU= 14.461
  05:27:53 STEP 2022 H=2022:00 +CPU= 23.349
  05:28:26 STEP 2023 H=2023:00 +CPU= 14.762
  05:29:00 STEP 2024 H=2024:00 +CPU= 14.752
  05:29:39 STEP 2025 H=2025:00 +CPU= 22.588
  05:30:07 STEP 2026 H=2026:00 +CPU= 14.916
  05:30:37 STEP 2027 H=2027:00 +CPU= 14.667
  05:31:32 STEP 2028 H=2028:00 +CPU= 25.806
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002028
  05:32:07 STEP 2029 H=2029:00 +CPU= 14.387
  05:32:41 STEP 2030 H=2030:00 +CPU= 14.952
  05:33:35 STEP 2031 H=2031:00 +CPU= 23.287
  05:34:13 STEP 2032 H=2032:00 +CPU= 14.991
  05:34:44 STEP 2033 H=2033:00 +CPU= 14.671
  05:35:45 STEP 2034 H=2034:00 +CPU= 23.781
  05:36:22 STEP 2035 H=2035:00 +CPU= 14.671
  05:36:55 STEP 2036 H=2036:00 +CPU= 15.163
  05:37:39 STEP 2037 H=2037:00 +CPU= 22.907
  05:38:02 STEP 2038 H=2038:00 +CPU= 14.465
  05:38:44 STEP 2039 H=2039:00 +CPU= 15.631
  05:39:40 STEP 2040 H=2040:00 +CPU= 27.322
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_84.zip
Uploading the intermediate file: upload_file_84.zip
  05:40:12 STEP 2041 H=2041:00 +CPU= 14.530
  05:40:49 STEP 2042 H=2042:00 +CPU= 15.084
  05:41:32 STEP 2043 H=2043:00 +CPU= 23.081
  05:42:02 STEP 2044 H=2044:00 +CPU= 14.886
  05:42:34 STEP 2045 H=2045:00 +CPU= 14.501
  05:43:26 STEP 2046 H=2046:00 +CPU= 23.596
  05:43:56 STEP 2047 H=2047:00 +CPU= 14.070
  05:44:34 STEP 2048 H=2048:00 +CPU= 15.563
  05:45:24 STEP 2049 H=2049:00 +CPU= 22.878
  05:45:59 STEP 2050 H=2050:00 +CPU= 15.029
  05:46:32 STEP 2051 H=2051:00 +CPU= 14.460
  05:47:27 STEP 2052 H=2052:00 +CPU= 25.845
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002052
  05:47:57 STEP 2053 H=2053:00 +CPU= 14.491
  05:48:39 STEP 2054 H=2054:00 +CPU= 15.179
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0940
(argv4) batchid: 960
(argv5) wuid: 12177584
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0c9f7be84c34f77f89e66967c57aacb7 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0940_1991050100_123_960_12177584.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0940_1991050100_123_960_12177584.zip
ic_ancil_file: ic_ancil_12177584
ifsdata_file: ifsdata_12177584
climate_data_file: clim_data_12177584
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_994106004c9c6922afdecae24eb0aaf8 to: /var/lib/boinc-client/slots/2/ic_ancil_12177584.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12177584.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_73fa1c00c70110052ef26e836db9d042 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12177584.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12177584.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12177584.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12177584.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12177584.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12177584.xml
-- Model is restarting --
Adjusting last_iter, 2055, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12177584.xml
last_cpu_time: 37260
upload_file_number: 85
last_iter: 2039
last_upload: 7344000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929
The child process has been launched with process id: 169995
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:169995:169995] [20230108:055013:1673128213:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:51:30 STEP 2040 H=2040:00 +CPU= 26.568
  05:52:01 STEP 2041 H=2041:00 +CPU= 14.458
  05:52:36 STEP 2042 H=2042:00 +CPU= 14.700
  05:53:29 STEP 2043 H=2043:00 +CPU= 23.391
  05:54:04 STEP 2044 H=2044:00 +CPU= 15.233
  05:54:35 STEP 2045 H=2045:00 +CPU= 14.837
  05:55:25 STEP 2046 H=2046:00 +CPU= 23.526
  05:55:58 STEP 2047 H=2047:00 +CPU= 14.377
  05:56:24 STEP 2048 H=2048:00 +CPU= 14.499
  05:57:06 STEP 2049 H=2049:00 +CPU= 23.039
  05:57:33 STEP 2050 H=2050:00 +CPU= 14.917
  05:58:15 STEP 2051 H=2051:00 +CPU= 14.971
  05:59:04 STEP 2052 H=2052:00 +CPU= 25.338
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002052
  05:59:35 STEP 2053 H=2053:00 +CPU= 14.398
  06:00:04 STEP 2054 H=2054:00 +CPU= 14.572
  06:00:56 STEP 2055 H=2055:00 +CPU= 23.673
  06:01:30 STEP 2056 H=2056:00 +CPU= 15.268
  06:02:04 STEP 2057 H=2057:00 +CPU= 14.387
  06:02:50 STEP 2058 H=2058:00 +CPU= 24.140
  06:03:23 STEP 2059 H=2059:00 +CPU= 14.807
  06:03:45 STEP 2060 H=2060:00 +CPU= 14.201
  06:04:33 STEP 2061 H=2061:00 +CPU= 23.140
  06:05:09 STEP 2062 H=2062:00 +CPU= 14.860
  06:05:48 STEP 2063 H=2063:00 +CPU= 15.354
  06:06:42 STEP 2064 H=2064:00 +CPU= 27.829
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_85.zip
Uploading the intermediate file: upload_file_85.zip
  06:07:14 STEP 2065 H=2065:00 +CPU= 14.366
  06:07:44 STEP 2066 H=2066:00 +CPU= 14.540
  06:08:34 STEP 2067 H=2067:00 +CPU= 23.350
  06:09:05 STEP 2068 H=2068:00 +CPU= 14.618
  06:09:36 STEP 2069 H=2069:00 +CPU= 14.640
  06:10:20 STEP 2070 H=2070:00 +CPU= 23.373
  06:10:47 STEP 2071 H=2071:00 +CPU= 14.440
  06:11:20 STEP 2072 H=2072:00 +CPU= 14.890
  06:11:56 STEP 2073 H=2073:00 +CPU= 22.290
  06:12:26 STEP 2074 H=2074:00 +CPU= 14.568
  06:12:54 STEP 2075 H=2075:00 +CPU= 14.402
  06:13:29 STEP 2076 H=2076:00 +CPU= 24.209
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002076
  06:13:59 STEP 2077 H=2077:00 +CPU= 14.294
  06:14:26 STEP 2078 H=2078:00 +CPU= 14.766
  06:15:02 STEP 2079 H=2079:00 +CPU= 22.302
  06:15:29 STEP 2080 H=2080:00 +CPU= 14.612
  06:15:51 STEP 2081 H=2081:00 +CPU= 13.930
  06:16:25 STEP 2082 H=2082:00 +CPU= 22.693
  06:16:48 STEP 2083 H=2083:00 +CPU= 14.167
  06:17:13 STEP 2084 H=2084:00 +CPU= 14.275
  06:17:48 STEP 2085 H=2085:00 +CPU= 22.462
  06:18:12 STEP 2086 H=2086:00 +CPU= 14.491
  06:18:38 STEP 2087 H=2087:00 +CPU= 14.744
  06:19:26 STEP 2088 H=2088:00 +CPU= 27.474
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_86.zip
Uploading the intermediate file: upload_file_86.zip
  06:19:49 STEP 2089 H=2089:00 +CPU= 14.430
  06:20:16 STEP 2090 H=2090:00 +CPU= 14.808
  06:20:51 STEP 2091 H=2091:00 +CPU= 22.470
  06:21:16 STEP 2092 H=2092:00 +CPU= 14.715
  06:21:41 STEP 2093 H=2093:00 +CPU= 14.167
  06:22:20 STEP 2094 H=2094:00 +CPU= 23.231
  06:22:41 STEP 2095 H=2095:00 +CPU= 14.338
  06:23:06 STEP 2096 H=2096:00 +CPU= 14.542
  06:23:43 STEP 2097 H=2097:00 +CPU= 22.645
  06:24:10 STEP 2098 H=2098:00 +CPU= 14.401
  06:24:34 STEP 2099 H=2099:00 +CPU= 14.334
  06:25:22 STEP 2100 H=2100:00 +CPU= 25.251
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002100
  06:25:50 STEP 2101 H=2101:00 +CPU= 14.396
  06:26:21 STEP 2102 H=2102:00 +CPU= 14.759
  06:27:02 STEP 2103 H=2103:00 +CPU= 22.726
  06:27:26 STEP 2104 H=2104:00 +CPU= 14.578
  06:27:50 STEP 2105 H=2105:00 +CPU= 14.227
  06:28:32 STEP 2106 H=2106:00 +CPU= 23.191
  06:28:59 STEP 2107 H=2107:00 +CPU= 14.291
  06:29:27 STEP 2108 H=2108:00 +CPU= 15.050
  06:30:06 STEP 2109 H=2109:00 +CPU= 22.873
  06:30:27 STEP 2110 H=2110:00 +CPU= 14.270
  06:30:57 STEP 2111 H=2111:00 +CPU= 14.881
  06:31:54 STEP 2112 H=2112:00 +CPU= 28.491
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_87.zip
Uploading the intermediate file: upload_file_87.zip
  06:32:19 STEP 2113 H=2113:00 +CPU= 14.228
  06:32:48 STEP 2114 H=2114:00 +CPU= 14.663
  06:33:33 STEP 2115 H=2115:00 +CPU= 23.418
  06:34:09 STEP 2116 H=2116:00 +CPU= 15.146
  06:34:40 STEP 2117 H=2117:00 +CPU= 14.119
  06:35:29 STEP 2118 H=2118:00 +CPU= 23.405
  06:36:02 STEP 2119 H=2119:00 +CPU= 14.499
  06:36:23 STEP 2120 H=2120:00 +CPU= 14.750
  06:36:58 STEP 2121 H=2121:00 +CPU= 22.412
  06:37:37 STEP 2122 H=2122:00 +CPU= 15.400
  06:38:08 STEP 2123 H=2123:00 +CPU= 14.674
  06:39:02 STEP 2124 H=2124:00 +CPU= 26.134
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002124
  06:39:35 STEP 2125 H=2125:00 +CPU= 15.031
  06:40:05 STEP 2126 H=2126:00 +CPU= 15.407
  06:40:54 STEP 2127 H=2127:00 +CPU= 23.109
  06:41:25 STEP 2128 H=2128:00 +CPU= 14.814
  06:41:55 STEP 2129 H=2129:00 +CPU= 14.173
  06:42:44 STEP 2130 H=2130:00 +CPU= 22.983
  06:43:18 STEP 2131 H=2131:00 +CPU= 14.809
  06:43:42 STEP 2132 H=2132:00 +CPU= 14.341
  06:44:30 STEP 2133 H=2133:00 +CPU= 22.850
  06:45:01 STEP 2134 H=2134:00 +CPU= 14.683
  06:45:33 STEP 2135 H=2135:00 +CPU= 14.784
  06:46:32 STEP 2136 H=2136:00 +CPU= 27.732
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_88.zip
Uploading the intermediate file: upload_file_88.zip
  06:46:51 STEP 2137 H=2137:00 +CPU= 14.258
  06:47:18 STEP 2138 H=2138:00 +CPU= 14.458
  06:48:07 STEP 2139 H=2139:00 +CPU= 23.247
  06:48:33 STEP 2140 H=2140:00 +CPU= 14.818
  06:49:03 STEP 2141 H=2141:00 +CPU= 14.103
  06:49:48 STEP 2142 H=2142:00 +CPU= 23.220
  06:50:15 STEP 2143 H=2143:00 +CPU= 14.674
  06:50:46 STEP 2144 H=2144:00 +CPU= 15.029
  06:51:32 STEP 2145 H=2145:00 +CPU= 23.182
  06:52:05 STEP 2146 H=2146:00 +CPU= 14.787
  06:52:41 STEP 2147 H=2147:00 +CPU= 15.006
  06:53:25 STEP 2148 H=2148:00 +CPU= 25.132
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002148
  06:53:59 STEP 2149 H=2149:00 +CPU= 14.901
  06:54:31 STEP 2150 H=2150:00 +CPU= 14.647
  06:55:11 STEP 2151 H=2151:00 +CPU= 23.152
  06:55:45 STEP 2152 H=2152:00 +CPU= 14.932
  06:56:14 STEP 2153 H=2153:00 +CPU= 14.720
  06:57:09 STEP 2154 H=2154:00 +CPU= 24.053
  06:57:37 STEP 2155 H=2155:00 +CPU= 14.450
  06:58:02 STEP 2156 H=2156:00 +CPU= 15.362
  06:58:48 STEP 2157 H=2157:00 +CPU= 22.890
  06:59:19 STEP 2158 H=2158:00 +CPU= 14.715
  06:59:52 STEP 2159 H=2159:00 +CPU= 15.761
  07:00:57 STEP 2160 H=2160:00 +CPU= 27.791
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_89.zip
Uploading the intermediate file: upload_file_89.zip
  07:01:23 STEP 2161 H=2161:00 +CPU= 14.284
  07:02:00 STEP 2162 H=2162:00 +CPU= 15.058
  07:02:49 STEP 2163 H=2163:00 +CPU= 23.553
  07:03:24 STEP 2164 H=2164:00 +CPU= 15.021
  07:03:52 STEP 2165 H=2165:00 +CPU= 14.505
  07:04:37 STEP 2166 H=2166:00 +CPU= 23.394
  07:05:07 STEP 2167 H=2167:00 +CPU= 14.874
  07:05:49 STEP 2168 H=2168:00 +CPU= 15.252
  07:06:37 STEP 2169 H=2169:00 +CPU= 23.540
  07:07:10 STEP 2170 H=2170:00 +CPU= 14.963
  07:07:48 STEP 2171 H=2171:00 +CPU= 14.882
  07:08:36 STEP 2172 H=2172:00 +CPU= 25.694
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002172
  07:09:12 STEP 2173 H=2173:00 +CPU= 14.943
  07:09:40 STEP 2174 H=2174:00 +CPU= 14.530
  07:10:27 STEP 2175 H=2175:00 +CPU= 23.442
  07:10:54 STEP 2176 H=2176:00 +CPU= 14.367
  07:11:27 STEP 2177 H=2177:00 +CPU= 15.104
  07:12:11 STEP 2178 H=2178:00 +CPU= 22.963
  07:12:40 STEP 2179 H=2179:00 +CPU= 14.386
  07:13:11 STEP 2180 H=2180:00 +CPU= 15.024
  07:14:04 STEP 2181 H=2181:00 +CPU= 22.607
  07:14:37 STEP 2182 H=2182:00 +CPU= 14.784
  07:15:10 STEP 2183 H=2183:00 +CPU= 14.849
  07:16:16 STEP 2184 H=2184:00 +CPU= 28.044
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_90.zip
Uploading the intermediate file: upload_file_90.zip
  07:16:51 STEP 2185 H=2185:00 +CPU= 14.692
  07:17:26 STEP 2186 H=2186:00 +CPU= 15.011
  07:18:16 STEP 2187 H=2187:00 +CPU= 23.494
  07:18:49 STEP 2188 H=2188:00 +CPU= 14.970
  07:19:21 STEP 2189 H=2189:00 +CPU= 14.709
  07:20:12 STEP 2190 H=2190:00 +CPU= 24.226
  07:20:37 STEP 2191 H=2191:00 +CPU= 14.662
  07:21:05 STEP 2192 H=2192:00 +CPU= 14.744
  07:21:59 STEP 2193 H=2193:00 +CPU= 23.621
  07:22:31 STEP 2194 H=2194:00 +CPU= 14.849
  07:23:03 STEP 2195 H=2195:00 +CPU= 14.545
  07:23:50 STEP 2196 H=2196:00 +CPU= 24.884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002196
  07:24:29 STEP 2197 H=2197:00 +CPU= 14.673
  07:25:02 STEP 2198 H=2198:00 +CPU= 14.916
  07:25:50 STEP 2199 H=2199:00 +CPU= 23.518
  07:26:25 STEP 2200 H=2200:00 +CPU= 14.674
  07:27:03 STEP 2201 H=2201:00 +CPU= 14.982
  07:27:51 STEP 2202 H=2202:00 +CPU= 23.595
  07:28:30 STEP 2203 H=2203:00 +CPU= 15.778
  07:28:53 STEP 2204 H=2204:00 +CPU= 14.341
  07:29:39 STEP 2205 H=2205:00 +CPU= 23.720
  07:30:10 STEP 2206 H=2206:00 +CPU= 15.340
  07:30:43 STEP 2207 H=2207:00 +CPU= 14.925
  07:31:50 STEP 2208 H=2208:00 +CPU= 28.357
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_91.zip
Uploading the intermediate file: upload_file_91.zip
  07:32:14 STEP 2209 H=2209:00 +CPU= 14.500
  07:32:46 STEP 2210 H=2210:00 +CPU= 15.094
  07:33:34 STEP 2211 H=2211:00 +CPU= 23.490
  07:34:06 STEP 2212 H=2212:00 +CPU= 14.909
  07:34:40 STEP 2213 H=2213:00 +CPU= 14.688
  07:35:18 STEP 2214 H=2214:00 +CPU= 22.658
  07:35:56 STEP 2215 H=2215:00 +CPU= 14.369
  07:36:27 STEP 2216 H=2216:00 +CPU= 15.067
  07:37:12 STEP 2217 H=2217:00 +CPU= 22.799
  07:37:45 STEP 2218 H=2218:00 +CPU= 14.929
  07:38:11 STEP 2219 H=2219:00 +CPU= 14.379
  07:38:55 STEP 2220 H=2220:00 +CPU= 25.519
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002220
  07:39:25 STEP 2221 H=2221:00 +CPU= 14.359
  07:40:00 STEP 2222 H=2222:00 +CPU= 14.891
  07:40:50 STEP 2223 H=2223:00 +CPU= 22.935
  07:41:26 STEP 2224 H=2224:00 +CPU= 14.929
  07:41:59 STEP 2225 H=2225:00 +CPU= 14.437
  07:42:50 STEP 2226 H=2226:00 +CPU= 24.621
  07:43:20 STEP 2227 H=2227:00 +CPU= 14.508
  07:43:48 STEP 2228 H=2228:00 +CPU= 14.486
  07:44:39 STEP 2229 H=2229:00 +CPU= 23.821
  07:45:13 STEP 2230 H=2230:00 +CPU= 14.897
  07:45:49 STEP 2231 H=2231:00 +CPU= 15.025
  07:46:47 STEP 2232 H=2232:00 +CPU= 28.365
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_92.zip
Uploading the intermediate file: upload_file_92.zip
  07:47:19 STEP 2233 H=2233:00 +CPU= 14.589
  07:47:51 STEP 2234 H=2234:00 +CPU= 14.856
  07:48:40 STEP 2235 H=2235:00 +CPU= 23.589
  07:49:12 STEP 2236 H=2236:00 +CPU= 15.245
  07:49:40 STEP 2237 H=2237:00 +CPU= 14.686
  07:50:26 STEP 2238 H=2238:00 +CPU= 23.551
  07:50:58 STEP 2239 H=2239:00 +CPU= 14.438
  07:51:25 STEP 2240 H=2240:00 +CPU= 14.545
  07:52:13 STEP 2241 H=2241:00 +CPU= 23.766
  07:52:45 STEP 2242 H=2242:00 +CPU= 14.737
  07:53:10 STEP 2243 H=2243:00 +CPU= 14.479
  07:54:03 STEP 2244 H=2244:00 +CPU= 25.530
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002244
  07:54:37 STEP 2245 H=2245:00 +CPU= 14.448
  07:55:09 STEP 2246 H=2246:00 +CPU= 14.541
  07:55:55 STEP 2247 H=2247:00 +CPU= 22.789
  07:56:28 STEP 2248 H=2248:00 +CPU= 15.386
  07:57:01 STEP 2249 H=2249:00 +CPU= 14.345
  07:57:49 STEP 2250 H=2250:00 +CPU= 23.755
  07:58:20 STEP 2251 H=2251:00 +CPU= 14.508
  07:58:50 STEP 2252 H=2252:00 +CPU= 14.700
  07:59:34 STEP 2253 H=2253:00 +CPU= 23.225
  08:00:03 STEP 2254 H=2254:00 +CPU= 14.656
  08:00:29 STEP 2255 H=2255:00 +CPU= 14.524
  08:01:31 STEP 2256 H=2256:00 +CPU= 27.825
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_93.zip
Uploading the intermediate file: upload_file_93.zip
  08:02:06 STEP 2257 H=2257:00 +CPU= 14.722
  08:02:40 STEP 2258 H=2258:00 +CPU= 15.352
  08:03:27 STEP 2259 H=2259:00 +CPU= 23.692
  08:03:49 STEP 2260 H=2260:00 +CPU= 14.372
  08:04:14 STEP 2261 H=2261:00 +CPU= 14.218
  08:04:58 STEP 2262 H=2262:00 +CPU= 22.726
  08:05:30 STEP 2263 H=2263:00 +CPU= 14.301
  08:06:04 STEP 2264 H=2264:00 +CPU= 14.810
  08:06:47 STEP 2265 H=2265:00 +CPU= 22.863
  08:07:19 STEP 2266 H=2266:00 +CPU= 15.039
  08:07:53 STEP 2267 H=2267:00 +CPU= 14.589
  08:08:47 STEP 2268 H=2268:00 +CPU= 25.443
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002268
  08:09:19 STEP 2269 H=2269:00 +CPU= 14.707
  08:09:49 STEP 2270 H=2270:00 +CPU= 14.491
  08:10:40 STEP 2271 H=2271:00 +CPU= 23.039
  08:11:14 STEP 2272 H=2272:00 +CPU= 14.745
  08:11:51 STEP 2273 H=2273:00 +CPU= 14.521
  08:12:36 STEP 2274 H=2274:00 +CPU= 24.137
  08:13:02 STEP 2275 H=2275:00 +CPU= 14.524
  08:13:30 STEP 2276 H=2276:00 +CPU= 14.843
  08:14:05 STEP 2277 H=2277:00 +CPU= 22.635
  08:14:32 STEP 2278 H=2278:00 +CPU= 14.714
  08:14:59 STEP 2279 H=2279:00 +CPU= 14.408
  08:15:43 STEP 2280 H=2280:00 +CPU= 27.045
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_94.zip
Uploading the intermediate file: upload_file_94.zip
  08:16:09 STEP 2281 H=2281:00 +CPU= 14.054
Uploading trickle at timestep: 8208000
  08:16:35 STEP 2282 H=2282:00 +CPU= 14.756
  08:17:08 STEP 2283 H=2283:00 +CPU= 22.331
  08:17:37 STEP 2284 H=2284:00 +CPU= 14.870
  08:18:01 STEP 2285 H=2285:00 +CPU= 14.266
  08:18:45 STEP 2286 H=2286:00 +CPU= 22.957
  08:19:06 STEP 2287 H=2287:00 +CPU= 14.011
  08:19:31 STEP 2288 H=2288:00 +CPU= 14.832
  08:20:08 STEP 2289 H=2289:00 +CPU= 22.667
  08:20:31 STEP 2290 H=2290:00 +CPU= 14.255
  08:20:54 STEP 2291 H=2291:00 +CPU= 14.201
  08:21:35 STEP 2292 H=2292:00 +CPU= 24.888
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002292
  08:21:58 STEP 2293 H=2293:00 +CPU= 14.895
  08:22:23 STEP 2294 H=2294:00 +CPU= 14.652
  08:23:01 STEP 2295 H=2295:00 +CPU= 22.523
  08:23:24 STEP 2296 H=2296:00 +CPU= 14.662
  08:23:47 STEP 2297 H=2297:00 +CPU= 14.160
  08:24:25 STEP 2298 H=2298:00 +CPU= 23.175
  08:24:51 STEP 2299 H=2299:00 +CPU= 14.216
  08:25:17 STEP 2300 H=2300:00 +CPU= 14.438
  08:25:57 STEP 2301 H=2301:00 +CPU= 22.404
  08:26:30 STEP 2302 H=2302:00 +CPU= 14.725
  08:26:54 STEP 2303 H=2303:00 +CPU= 14.717
  08:27:52 STEP 2304 H=2304:00 +CPU= 27.504
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_95.zip
Uploading the intermediate file: upload_file_95.zip
  08:28:24 STEP 2305 H=2305:00 +CPU= 14.424
  08:28:55 STEP 2306 H=2306:00 +CPU= 14.339
  08:29:36 STEP 2307 H=2307:00 +CPU= 22.762
  08:30:05 STEP 2308 H=2308:00 +CPU= 14.865
  08:30:32 STEP 2309 H=2309:00 +CPU= 14.095
  08:31:14 STEP 2310 H=2310:00 +CPU= 22.698
  08:31:44 STEP 2311 H=2311:00 +CPU= 14.484
  08:32:10 STEP 2312 H=2312:00 +CPU= 14.182
  08:32:52 STEP 2313 H=2313:00 +CPU= 22.584
  08:33:14 STEP 2314 H=2314:00 +CPU= 14.731
  08:33:44 STEP 2315 H=2315:00 +CPU= 14.188
  08:34:30 STEP 2316 H=2316:00 +CPU= 25.018
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002316
  08:34:55 STEP 2317 H=2317:00 +CPU= 14.101
  08:35:23 STEP 2318 H=2318:00 +CPU= 14.522
  08:36:00 STEP 2319 H=2319:00 +CPU= 22.911
  08:36:35 STEP 2320 H=2320:00 +CPU= 15.018
  08:37:01 STEP 2321 H=2321:00 +CPU= 14.388
  08:37:54 STEP 2322 H=2322:00 +CPU= 24.023
  08:38:30 STEP 2323 H=2323:00 +CPU= 14.678
  08:39:01 STEP 2324 H=2324:00 +CPU= 15.064
  08:39:57 STEP 2325 H=2325:00 +CPU= 23.600
  08:40:31 STEP 2326 H=2326:00 +CPU= 14.834
  08:41:05 STEP 2327 H=2327:00 +CPU= 14.669
  08:41:56 STEP 2328 H=2328:00 +CPU= 27.710
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_96.zip
Uploading the intermediate file: upload_file_96.zip
  08:42:27 STEP 2329 H=2329:00 +CPU= 14.607
  08:43:07 STEP 2330 H=2330:00 +CPU= 15.376
  08:43:53 STEP 2331 H=2331:00 +CPU= 23.001
  08:44:21 STEP 2332 H=2332:00 +CPU= 14.635
  08:44:56 STEP 2333 H=2333:00 +CPU= 14.488
  08:45:46 STEP 2334 H=2334:00 +CPU= 23.775
  08:46:20 STEP 2335 H=2335:00 +CPU= 14.480
  08:46:52 STEP 2336 H=2336:00 +CPU= 15.201
  08:47:52 STEP 2337 H=2337:00 +CPU= 23.589
  08:48:22 STEP 2338 H=2338:00 +CPU= 14.820
  08:48:59 STEP 2339 H=2339:00 +CPU= 15.021
  08:49:55 STEP 2340 H=2340:00 +CPU= 25.446
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002340
  08:50:26 STEP 2341 H=2341:00 +CPU= 14.372
  08:50:55 STEP 2342 H=2342:00 +CPU= 14.826
  08:51:43 STEP 2343 H=2343:00 +CPU= 22.513
  08:52:16 STEP 2344 H=2344:00 +CPU= 14.827
  08:52:50 STEP 2345 H=2345:00 +CPU= 15.173
  08:53:38 STEP 2346 H=2346:00 +CPU= 23.470
  08:54:07 STEP 2347 H=2347:00 +CPU= 14.217
  08:54:44 STEP 2348 H=2348:00 +CPU= 15.239
  08:55:31 STEP 2349 H=2349:00 +CPU= 23.169
  08:56:00 STEP 2350 H=2350:00 +CPU= 15.121
  08:56:32 STEP 2351 H=2351:00 +CPU= 15.196
  08:57:37 STEP 2352 H=2352:00 +CPU= 28.302
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_97.zip
Uploading the intermediate file: upload_file_97.zip
  08:58:13 STEP 2353 H=2353:00 +CPU= 14.707
  08:58:42 STEP 2354 H=2354:00 +CPU= 15.171
  08:59:31 STEP 2355 H=2355:00 +CPU= 22.874
  09:00:07 STEP 2356 H=2356:00 +CPU= 15.112
  09:00:30 STEP 2357 H=2357:00 +CPU= 14.244
  09:01:21 STEP 2358 H=2358:00 +CPU= 23.886
  09:01:56 STEP 2359 H=2359:00 +CPU= 15.257
  09:02:35 STEP 2360 H=2360:00 +CPU= 15.219
  09:03:39 STEP 2361 H=2361:00 +CPU= 24.344
  09:04:10 STEP 2362 H=2362:00 +CPU= 15.073
  09:04:43 STEP 2363 H=2363:00 +CPU= 14.680
  09:05:40 STEP 2364 H=2364:00 +CPU= 26.285
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002364
  09:06:12 STEP 2365 H=2365:00 +CPU= 14.545
  09:06:54 STEP 2366 H=2366:00 +CPU= 15.612
  09:07:45 STEP 2367 H=2367:00 +CPU= 22.565
  09:08:12 STEP 2368 H=2368:00 +CPU= 14.469
  09:08:46 STEP 2369 H=2369:00 +CPU= 14.462
  09:09:30 STEP 2370 H=2370:00 +CPU= 23.584
  09:10:02 STEP 2371 H=2371:00 +CPU= 14.679
  09:10:38 STEP 2372 H=2372:00 +CPU= 14.837
  09:11:26 STEP 2373 H=2373:00 +CPU= 22.655
  09:12:03 STEP 2374 H=2374:00 +CPU= 15.756
  09:12:28 STEP 2375 H=2375:00 +CPU= 14.292
  09:13:24 STEP 2376 H=2376:00 +CPU= 28.041
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_98.zip
Uploading the intermediate file: upload_file_98.zip
  09:13:55 STEP 2377 H=2377:00 +CPU= 14.269
  09:14:24 STEP 2378 H=2378:00 +CPU= 14.523
  09:15:12 STEP 2379 H=2379:00 +CPU= 23.566
  09:15:46 STEP 2380 H=2380:00 +CPU= 15.508
  09:16:15 STEP 2381 H=2381:00 +CPU= 14.873
  09:17:05 STEP 2382 H=2382:00 +CPU= 23.586
  09:17:30 STEP 2383 H=2383:00 +CPU= 14.245
  09:18:01 STEP 2384 H=2384:00 +CPU= 15.090
  09:18:45 STEP 2385 H=2385:00 +CPU= 23.270
  09:19:17 STEP 2386 H=2386:00 +CPU= 14.487
  09:19:47 STEP 2387 H=2387:00 +CPU= 14.357
  09:20:40 STEP 2388 H=2388:00 +CPU= 26.006
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002388
  09:21:08 STEP 2389 H=2389:00 +CPU= 14.523
  09:21:42 STEP 2390 H=2390:00 +CPU= 15.236
  09:22:35 STEP 2391 H=2391:00 +CPU= 23.208
  09:23:08 STEP 2392 H=2392:00 +CPU= 15.158
  09:23:44 STEP 2393 H=2393:00 +CPU= 14.484
  09:24:34 STEP 2394 H=2394:00 +CPU= 23.852
  09:25:17 STEP 2395 H=2395:00 +CPU= 14.787
  09:25:44 STEP 2396 H=2396:00 +CPU= 15.255
  09:26:34 STEP 2397 H=2397:00 +CPU= 23.449
  09:27:03 STEP 2398 H=2398:00 +CPU= 14.529
  09:27:32 STEP 2399 H=2399:00 +CPU= 14.760
  09:28:33 STEP 2400 H=2400:00 +CPU= 28.199
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_99.zip
Uploading the intermediate file: upload_file_99.zip
  09:29:15 STEP 2401 H=2401:00 +CPU= 14.278
  09:29:47 STEP 2402 H=2402:00 +CPU= 14.368
  09:30:27 STEP 2403 H=2403:00 +CPU= 22.706
  09:31:08 STEP 2404 H=2404:00 +CPU= 15.758
  09:31:40 STEP 2405 H=2405:00 +CPU= 14.819
  09:32:25 STEP 2406 H=2406:00 +CPU= 23.521
  09:32:56 STEP 2407 H=2407:00 +CPU= 14.342
  09:33:29 STEP 2408 H=2408:00 +CPU= 14.596
  09:34:18 STEP 2409 H=2409:00 +CPU= 22.863
  09:34:51 STEP 2410 H=2410:00 +CPU= 14.806
  09:35:24 STEP 2411 H=2411:00 +CPU= 14.338
  09:36:29 STEP 2412 H=2412:00 +CPU= 26.232
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002412
  09:36:53 STEP 2413 H=2413:00 +CPU= 14.160
  09:37:23 STEP 2414 H=2414:00 +CPU= 14.783
  09:38:15 STEP 2415 H=2415:00 +CPU= 23.213
  09:38:48 STEP 2416 H=2416:00 +CPU= 15.188
  09:39:23 STEP 2417 H=2417:00 +CPU= 14.766
  09:40:10 STEP 2418 H=2418:00 +CPU= 23.379
  09:40:39 STEP 2419 H=2419:00 +CPU= 14.274
  09:41:07 STEP 2420 H=2420:00 +CPU= 14.719
  09:41:53 STEP 2421 H=2421:00 +CPU= 23.311
  09:42:22 STEP 2422 H=2422:00 +CPU= 14.344
  09:42:58 STEP 2423 H=2423:00 +CPU= 15.704
  09:43:59 STEP 2424 H=2424:00 +CPU= 27.907
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929_100.zip
Uploading the intermediate file: upload_file_100.zip
  09:44:34 STEP 2425 H=2425:00 +CPU= 14.210
  09:45:11 STEP 2426 H=2426:00 +CPU= 15.140
  09:46:00 STEP 2427 H=2427:00 +CPU= 22.973
  09:46:23 STEP 2428 H=2428:00 +CPU= 14.096
  09:46:51 STEP 2429 H=2429:00 +CPU= 14.728
  09:47:36 STEP 2430 H=2430:00 +CPU= 23.018
  09:48:02 STEP 2431 H=2431:00 +CPU= 13.985
  09:48:36 STEP 2432 H=2432:00 +CPU= 14.914
  09:49:15 STEP 2433 H=2433:00 +CPU= 22.422
  09:49:46 STEP 2434 H=2434:00 +CPU= 14.676
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0940
(argv4) batchid: 960
(argv5) wuid: 12177584
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177584
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0c9f7be84c34f77f89e66967c57aacb7 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0940_1991050100_123_960_12177584.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0940_1991050100_123_960_12177584.zip
ic_ancil_file: ic_ancil_12177584
ifsdata_file: ifsdata_12177584
climate_data_file: clim_data_12177584
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_994106004c9c6922afdecae24eb0aaf8 to: /var/lib/boinc-client/slots/2/ic_ancil_12177584.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12177584.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_73fa1c00c70110052ef26e836db9d042 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12177584.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12177584.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12177584.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12177584.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12177584.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12177584.xml
-- Model is restarting --
Adjusting last_iter, 2435, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12177584.xml
last_cpu_time: 44305
upload_file_number: 101
last_iter: 2423
last_upload: 8726400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0940_1991050100_123_960_12177584_0_r1605727929
The child process has been launched with process id: 185746
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:server0:1:1:185746:185746] [20230108:095115:1673142675:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:52:20 STEP 2424 H=2424:00 +CPU= 25.657
  09:52:54 STEP 2425 H=2425:00 +CPU= 14.835
  09:53:28 STEP 2426 H=2426:00 +CPU= 15.008
  09:54:08 STEP 2427 H=2427:00 +CPU= 23.179
  09:54:45 STEP 2428 H=2428:00 +CPU= 15.314
  09:55:24 STEP 2429 H=2429:00 +CPU= 14.834
  09:56:22 STEP 2430 H=2430:00 +CPU= 24.178
  09:56:53 STEP 2431 H=2431:00 +CPU= 14.196
  09:57:27 STEP 2432 H=2432:00 +CPU= 14.841
  09:58:24 STEP 2433 H=2433:00 +CPU= 23.280
  09:58:49 STEP 2434 H=2434:00 +CPU= 14.304
  09:59:19 STEP 2435 H=2435:00 +CPU= 14.128
  10:00:08 STEP 2436 H=2436:00 +CPU= 25.570
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002436
  10:00:32 STEP 2437 H=2437:00 +CPU= 14.125
  10:01:07 STEP 2438 H=2438:00 +CPU= 15.014
  10:01:47 STEP 2439 H=2439:00 +CPU= 22.886
  10:02:20 STEP 2440 H=2440:00 +CPU= 15.374
  10:02:56 STEP 2441 H=2441:00 +CPU= 15.010
  10:03:36 STEP 2442 H=2442:00 +CPU= 23.140
  10:04:06 STEP 2443 H=2443:00 +CPU= 14.823
  10:04:34 STEP 2444 H=2444:00 +CPU= 14.368
  10:05:23 STEP 2445 H=2445:00 +CPU= 22.975
  10:05:56 STEP 2446 H=2446:00 +CPU= 15.475
  10:06:23 STEP 2447 H=2447:00 +CPU= 14.813
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:15 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 8,208,000 41,570 0.0051
09 May 2023 16:12:04 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 6,825,600 34,461 0.0050
09 May 2023 16:11:53 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 5,184,000 26,318 0.0051
09 May 2023 16:11:42 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 3,715,200 19,310 0.0052
09 May 2023 16:11:30 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 2,246,400 11,999 0.0053
09 May 2023 16:11:18 1536968 22262383 oifs_43r3_ps_0940_1991050100_123_960_12177584_0 864,000 5,010 0.0058


©2024 climateprediction.net