climateprediction.net home page
Task 22248712

Task 22248712

Name oifs_43r3_ps_1554_2021050100_123_946_12164643_1
Workunit 12164643
Created 30 Nov 2022, 22:44:32 UTC
Sent 30 Nov 2022, 22:45:40 UTC
Report deadline 30 Dec 2022, 22:45:40 UTC
Received 1 Dec 2022, 12:30:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1527784
Run time 3 hours 0 min 52 sec
CPU time 1 hours 52 min 21 sec
Validate state Invalid
Credit 3,024.00
Device peak FLOPS 6.65 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.01
x86_64-pc-linux-gnu
Peak working set size 4,418.29 MB
Peak swap size 4,978.06 MB
Peak disk usage 2,126.66 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
ermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_6.zip
Uploading the intermediate file: upload_file_6.zip
  23:30:29 STEP  169 H= 169:00 +CPU= 12.930
  23:30:43 STEP  170 H= 170:00 +CPU= 13.317
Uploading trickle at timestep: 604800
  23:31:03 STEP  171 H= 171:00 +CPU= 20.304
  23:31:17 STEP  172 H= 172:00 +CPU= 13.351
  23:31:30 STEP  173 H= 173:00 +CPU= 12.991
  23:31:51 STEP  174 H= 174:00 +CPU= 20.751
  23:32:04 STEP  175 H= 175:00 +CPU= 12.967
  23:32:17 STEP  176 H= 176:00 +CPU= 13.377
  23:32:38 STEP  177 H= 177:00 +CPU= 20.334
  23:32:51 STEP  178 H= 178:00 +CPU= 13.316
  23:33:04 STEP  179 H= 179:00 +CPU= 12.993
  23:33:27 STEP  180 H= 180:00 +CPU= 22.538
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000180
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000180
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000180
  23:33:40 STEP  181 H= 181:00 +CPU= 12.994
  23:33:53 STEP  182 H= 182:00 +CPU= 13.379
  23:34:14 STEP  183 H= 183:00 +CPU= 20.350
  23:34:27 STEP  184 H= 184:00 +CPU= 13.344
  23:34:40 STEP  185 H= 185:00 +CPU= 12.965
  23:35:01 STEP  186 H= 186:00 +CPU= 20.722
  23:35:14 STEP  187 H= 187:00 +CPU= 12.975
  23:35:27 STEP  188 H= 188:00 +CPU= 13.348
  23:35:48 STEP  189 H= 189:00 +CPU= 20.293
  23:36:01 STEP  190 H= 190:00 +CPU= 13.359
  23:36:15 STEP  191 H= 191:00 +CPU= 13.359
  23:36:39 STEP  192 H= 192:00 +CPU= 24.233
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000192
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000192
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000168
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000180
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000180
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_7.zip
Uploading the intermediate file: upload_file_7.zip
  23:36:52 STEP  193 H= 193:00 +CPU= 12.959
  23:37:05 STEP  194 H= 194:00 +CPU= 13.310
Uploading trickle at timestep: 691200
  23:37:26 STEP  195 H= 195:00 +CPU= 20.306
  23:37:39 STEP  196 H= 196:00 +CPU= 13.326
  23:37:52 STEP  197 H= 197:00 +CPU= 12.966
  23:38:13 STEP  198 H= 198:00 +CPU= 20.752
  23:38:26 STEP  199 H= 199:00 +CPU= 12.974
  23:38:40 STEP  200 H= 200:00 +CPU= 13.356
  23:39:00 STEP  201 H= 201:00 +CPU= 20.379
  23:39:13 STEP  202 H= 202:00 +CPU= 13.383
  23:39:27 STEP  203 H= 203:00 +CPU= 13.039
  23:39:49 STEP  204 H= 204:00 +CPU= 22.463
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000204
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000204
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000204
  23:40:02 STEP  205 H= 205:00 +CPU= 12.966
  23:40:16 STEP  206 H= 206:00 +CPU= 13.355
  23:40:36 STEP  207 H= 207:00 +CPU= 20.348
  23:40:49 STEP  208 H= 208:00 +CPU= 13.364
  23:41:02 STEP  209 H= 209:00 +CPU= 12.997
  23:41:23 STEP  210 H= 210:00 +CPU= 20.785
  23:41:36 STEP  211 H= 211:00 +CPU= 12.963
  23:41:50 STEP  212 H= 212:00 +CPU= 13.303
  23:42:10 STEP  213 H= 213:00 +CPU= 20.299
  23:42:24 STEP  214 H= 214:00 +CPU= 13.321
  23:42:38 STEP  215 H= 215:00 +CPU= 13.360
  23:43:04 STEP  216 H= 216:00 +CPU= 24.509
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000216
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000216
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000204
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000204
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_8.zip
Uploading the intermediate file: upload_file_8.zip
  23:43:18 STEP  217 H= 217:00 +CPU= 13.124
  23:43:31 STEP  218 H= 218:00 +CPU= 13.379
Uploading trickle at timestep: 777600
  23:43:52 STEP  219 H= 219:00 +CPU= 20.415
  23:44:06 STEP  220 H= 220:00 +CPU= 13.390
  23:44:19 STEP  221 H= 221:00 +CPU= 13.096
  23:44:41 STEP  222 H= 222:00 +CPU= 21.013
  23:44:55 STEP  223 H= 223:00 +CPU= 13.143
  23:45:09 STEP  224 H= 224:00 +CPU= 13.501
  23:45:29 STEP  225 H= 225:00 +CPU= 20.284
  23:45:43 STEP  226 H= 226:00 +CPU= 13.396
  23:45:56 STEP  227 H= 227:00 +CPU= 13.065
  23:46:18 STEP  228 H= 228:00 +CPU= 22.422
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000228
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000228
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000228
  23:46:31 STEP  229 H= 229:00 +CPU= 12.995
  23:46:45 STEP  230 H= 230:00 +CPU= 13.444
  23:47:05 STEP  231 H= 231:00 +CPU= 20.416
  23:47:19 STEP  232 H= 232:00 +CPU= 13.342
  23:47:32 STEP  233 H= 233:00 +CPU= 13.098
  23:47:53 STEP  234 H= 234:00 +CPU= 20.783
  23:48:07 STEP  235 H= 235:00 +CPU= 12.989
  23:48:20 STEP  236 H= 236:00 +CPU= 13.370
  23:48:42 STEP  237 H= 237:00 +CPU= 20.564
  23:48:56 STEP  238 H= 238:00 +CPU= 13.370
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4745MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4745 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LARCINA 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234914:1669852154:3794.638] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  LASCAW 
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234915:1669852155:3795.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:45012:45012] [20221130:234915:1669852155:3795.638] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
master.exe         0000000001CE916B  Unknown               Unknown  Unknown
master.exe         000000000081FFF1  Unknown               Unknown  Unknown
master.exe         0000000001DC9090  Unknown               Unknown  Unknown
master.exe         0000000001D53DA4  Unknown               Unknown  Unknown
master.exe         0000000000DF1251  lascaw_                   470  lascaw.F90
master.exe         0000000000DED525  larcina_                  319  larcina.F90
master.exe         0000000000DEABF8  larmes_                   603  larmes.F90
master.exe         0000000000DE50FD  lapinea_                  270  lapinea.F90
master.exe         0000000000D7A4EC  call_sl_                  300  call_sl.F90
master.exe         0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
master.exe         0000000000BF9958  gp_model_                 533  gp_model.F90
master.exe         00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
master.exe         0000000000BBC1CE  scan2m_                   535  scan2m.F90
master.exe         000000000057A664  stepo_                    327  stepo.F90
master.exe         000000000055DEEC  cnt4_                    1133  cnt4.F90
master.exe         00000000005471C9  cnt3_                     267  cnt3.F90
master.exe         0000000000546412  cnt2_                      88  cnt2.F90
master.exe         0000000000545E28  cnt1_                      92  cnt1.F90
master.exe         000000000040708D  cnt0_                     146  cnt0.F90
master.exe         000000000040220F  MAIN__                     96  master.F90
master.exe         00000000004021A2  Unknown               Unknown  Unknown
master.exe         0000000001DCA390  Unknown               Unknown  Unknown
master.exe         000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 1554
(argv4) batchid: 946
(argv5) wuid: 12164643
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7efd10e034b9da91f94cb5de87cadaeb to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_1554_2021050100_123_946_12164643.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_1554_2021050100_123_946_12164643.zip
ic_ancil_file: ic_ancil_12164643
ifsdata_file: ifsdata_12164643
climate_data_file: clim_data_12164643
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_9255127d70349c16abe4c44712e090a6 to: /var/lib/boinc-client/slots/6/ic_ancil_12164643.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12164643.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f4094c751d09804c28bd45de9b9d17e8 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12164643.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12164643.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12164643.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12164643.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708
The child process has been launched with process id: 14278
Executing the command: /var/lib/boinc-client/slots/6/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antony-OptiPlex-3060:1:1:14278:14278] [20221201:102206:1669890126:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:22:33 STEP  216 H= 216:00 +CPU= 23.379
  10:22:47 STEP  217 H= 217:00 +CPU= 13.241
  10:23:01 STEP  218 H= 218:00 +CPU= 13.569
  10:23:22 STEP  219 H= 219:00 +CPU= 20.692
  10:23:35 STEP  220 H= 220:00 +CPU= 13.517
  10:23:49 STEP  221 H= 221:00 +CPU= 13.303
  10:24:11 STEP  222 H= 222:00 +CPU= 21.228
  10:24:25 STEP  223 H= 223:00 +CPU= 13.413
  10:24:39 STEP  224 H= 224:00 +CPU= 13.627
  10:25:00 STEP  225 H= 225:00 +CPU= 20.691
  10:25:14 STEP  226 H= 226:00 +CPU= 13.693
  10:25:27 STEP  227 H= 227:00 +CPU= 13.212
  10:25:50 STEP  228 H= 228:00 +CPU= 22.779
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000228
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000228
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000228
  10:26:04 STEP  229 H= 229:00 +CPU= 13.214
  10:26:18 STEP  230 H= 230:00 +CPU= 13.674
  10:26:38 STEP  231 H= 231:00 +CPU= 20.632
  10:26:52 STEP  232 H= 232:00 +CPU= 13.562
  10:27:06 STEP  233 H= 233:00 +CPU= 13.182
  10:27:27 STEP  234 H= 234:00 +CPU= 21.064
  10:27:40 STEP  235 H= 235:00 +CPU= 13.239
  10:27:54 STEP  236 H= 236:00 +CPU= 13.613
  10:28:15 STEP  237 H= 237:00 +CPU= 20.632
  10:28:29 STEP  238 H= 238:00 +CPU= 13.552
  10:28:43 STEP  239 H= 239:00 +CPU= 13.617
  10:29:07 STEP  240 H= 240:00 +CPU= 24.432
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000240
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000240
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000228
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_1.zip
Uploading the intermediate file: upload_file_1.zip
  10:29:21 STEP  241 H= 241:00 +CPU= 13.203
Uploading trickle at timestep: 864000
  10:29:35 STEP  242 H= 242:00 +CPU= 13.545
  10:29:55 STEP  243 H= 243:00 +CPU= 20.658
  10:30:09 STEP  244 H= 244:00 +CPU= 13.562
  10:30:22 STEP  245 H= 245:00 +CPU= 13.207
  10:30:44 STEP  246 H= 246:00 +CPU= 21.083
  10:30:57 STEP  247 H= 247:00 +CPU= 13.231
  10:31:11 STEP  248 H= 248:00 +CPU= 13.625
  10:31:32 STEP  249 H= 249:00 +CPU= 20.669
  10:31:45 STEP  250 H= 250:00 +CPU= 13.626
  10:31:59 STEP  251 H= 251:00 +CPU= 13.240
  10:32:22 STEP  252 H= 252:00 +CPU= 22.848
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000252
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000252
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000252
  10:32:35 STEP  253 H= 253:00 +CPU= 13.239
  10:32:49 STEP  254 H= 254:00 +CPU= 13.613
  10:33:10 STEP  255 H= 255:00 +CPU= 20.657
  10:33:24 STEP  256 H= 256:00 +CPU= 13.589
  10:33:37 STEP  257 H= 257:00 +CPU= 13.240
  10:33:58 STEP  258 H= 258:00 +CPU= 21.026
  10:34:12 STEP  259 H= 259:00 +CPU= 13.248
  10:34:25 STEP  260 H= 260:00 +CPU= 13.565
  10:34:46 STEP  261 H= 261:00 +CPU= 20.634
  10:35:00 STEP  262 H= 262:00 +CPU= 13.619
  10:35:14 STEP  263 H= 263:00 +CPU= 13.748
  10:35:40 STEP  264 H= 264:00 +CPU= 24.838
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000264
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000264
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_2.zip
Uploading the intermediate file: upload_file_2.zip
  10:35:54 STEP  265 H= 265:00 +CPU= 13.312
  10:36:08 STEP  266 H= 266:00 +CPU= 13.684
Uploading trickle at timestep: 950400
  10:36:29 STEP  267 H= 267:00 +CPU= 20.731
  10:36:43 STEP  268 H= 268:00 +CPU= 13.628
  10:36:57 STEP  269 H= 269:00 +CPU= 13.277
  10:37:19 STEP  270 H= 270:00 +CPU= 21.184
  10:37:32 STEP  271 H= 271:00 +CPU= 13.307
  10:37:46 STEP  272 H= 272:00 +CPU= 13.701
  10:38:08 STEP  273 H= 273:00 +CPU= 20.757
  10:38:22 STEP  274 H= 274:00 +CPU= 13.657
  10:38:35 STEP  275 H= 275:00 +CPU= 13.320
  10:38:59 STEP  276 H= 276:00 +CPU= 22.982
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000276
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000276
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000276
  10:39:13 STEP  277 H= 277:00 +CPU= 13.293
  10:39:26 STEP  278 H= 278:00 +CPU= 13.662
  10:39:48 STEP  279 H= 279:00 +CPU= 20.801
  10:40:02 STEP  280 H= 280:00 +CPU= 13.703
  10:40:15 STEP  281 H= 281:00 +CPU= 13.216
  10:40:37 STEP  282 H= 282:00 +CPU= 21.286
  10:40:51 STEP  283 H= 283:00 +CPU= 13.353
  10:41:07 STEP  284 H= 284:00 +CPU= 14.196
  10:41:29 STEP  285 H= 285:00 +CPU= 21.121
  10:41:43 STEP  286 H= 286:00 +CPU= 13.964
  10:41:58 STEP  287 H= 287:00 +CPU= 14.125
  10:42:24 STEP  288 H= 288:00 +CPU= 25.077
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000288
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000288
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000276
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_3.zip
Uploading the intermediate file: upload_file_3.zip
  10:42:38 STEP  289 H= 289:00 +CPU= 13.618
  10:42:53 STEP  290 H= 290:00 +CPU= 14.019
Uploading trickle at timestep: 1036800
  10:43:14 STEP  291 H= 291:00 +CPU= 21.003
  10:43:28 STEP  292 H= 292:00 +CPU= 13.853
  10:43:42 STEP  293 H= 293:00 +CPU= 13.521
  10:44:04 STEP  294 H= 294:00 +CPU= 21.436
  10:44:17 STEP  295 H= 295:00 +CPU= 13.501
  10:44:31 STEP  296 H= 296:00 +CPU= 13.839
  10:44:53 STEP  297 H= 297:00 +CPU= 20.999
  10:45:07 STEP  298 H= 298:00 +CPU= 13.845
  10:45:20 STEP  299 H= 299:00 +CPU= 13.484
  10:45:44 STEP  300 H= 300:00 +CPU= 23.153
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000300
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000300
  10:45:58 STEP  301 H= 301:00 +CPU= 13.496
  10:46:12 STEP  302 H= 302:00 +CPU= 13.941
  10:46:33 STEP  303 H= 303:00 +CPU= 21.018
  10:46:47 STEP  304 H= 304:00 +CPU= 13.855
  10:47:01 STEP  305 H= 305:00 +CPU= 13.536
  10:47:23 STEP  306 H= 306:00 +CPU= 21.466
  10:47:37 STEP  307 H= 307:00 +CPU= 13.511
  10:47:51 STEP  308 H= 308:00 +CPU= 13.868
  10:48:12 STEP  309 H= 309:00 +CPU= 20.956
  10:48:26 STEP  310 H= 310:00 +CPU= 13.824
  10:48:40 STEP  311 H= 311:00 +CPU= 13.908
  10:49:05 STEP  312 H= 312:00 +CPU= 24.968
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000312
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000300
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_4.zip
Uploading the intermediate file: upload_file_4.zip
  10:49:19 STEP  313 H= 313:00 +CPU= 13.479
  10:49:33 STEP  314 H= 314:00 +CPU= 13.893
Uploading trickle at timestep: 1123200
  10:49:54 STEP  315 H= 315:00 +CPU= 21.027
  10:50:08 STEP  316 H= 316:00 +CPU= 13.904
  10:50:22 STEP  317 H= 317:00 +CPU= 13.517
  10:50:44 STEP  318 H= 318:00 +CPU= 21.453
  10:50:58 STEP  319 H= 319:00 +CPU= 13.569
  10:51:12 STEP  320 H= 320:00 +CPU= 13.916
  10:51:33 STEP  321 H= 321:00 +CPU= 21.040
  10:51:47 STEP  322 H= 322:00 +CPU= 13.907
  10:52:01 STEP  323 H= 323:00 +CPU= 13.584
  10:52:25 STEP  324 H= 324:00 +CPU= 23.290
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000324
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000324
  10:52:38 STEP  325 H= 325:00 +CPU= 13.533
  10:52:53 STEP  326 H= 326:00 +CPU= 14.116
  10:53:15 STEP  327 H= 327:00 +CPU= 21.189
  10:53:29 STEP  328 H= 328:00 +CPU= 13.909
  10:53:43 STEP  329 H= 329:00 +CPU= 13.622
  10:54:05 STEP  330 H= 330:00 +CPU= 21.627
  10:54:18 STEP  331 H= 331:00 +CPU= 13.550
  10:54:33 STEP  332 H= 332:00 +CPU= 14.069
  10:54:55 STEP  333 H= 333:00 +CPU= 21.249
  10:55:09 STEP  334 H= 334:00 +CPU= 13.995
  10:55:24 STEP  335 H= 335:00 +CPU= 14.059
  10:55:49 STEP  336 H= 336:00 +CPU= 25.121
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000336
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000324
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_5.zip
Uploading the intermediate file: upload_file_5.zip
  10:56:03 STEP  337 H= 337:00 +CPU= 13.529
  10:56:18 STEP  338 H= 338:00 +CPU= 13.940
Uploading trickle at timestep: 1209600
  10:56:39 STEP  339 H= 339:00 +CPU= 20.970
  10:56:53 STEP  340 H= 340:00 +CPU= 13.922
  10:57:06 STEP  341 H= 341:00 +CPU= 13.557
  10:57:28 STEP  342 H= 342:00 +CPU= 21.409
  10:57:41 STEP  343 H= 343:00 +CPU= 13.490
  10:57:55 STEP  344 H= 344:00 +CPU= 13.881
  10:58:16 STEP  345 H= 345:00 +CPU= 20.993
  10:58:30 STEP  346 H= 346:00 +CPU= 13.899
  10:58:44 STEP  347 H= 347:00 +CPU= 13.559
  10:59:07 STEP  348 H= 348:00 +CPU= 23.183
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000348
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000348
  10:59:21 STEP  349 H= 349:00 +CPU= 13.533
  10:59:35 STEP  350 H= 350:00 +CPU= 13.897
  10:59:56 STEP  351 H= 351:00 +CPU= 21.069
  11:00:10 STEP  352 H= 352:00 +CPU= 13.885
  11:00:24 STEP  353 H= 353:00 +CPU= 13.479
  11:00:45 STEP  354 H= 354:00 +CPU= 21.427
  11:00:59 STEP  355 H= 355:00 +CPU= 13.451
  11:01:13 STEP  356 H= 356:00 +CPU= 13.824
  11:01:34 STEP  357 H= 357:00 +CPU= 21.015
  11:01:48 STEP  358 H= 358:00 +CPU= 13.810
  11:02:02 STEP  359 H= 359:00 +CPU= 13.874
  11:02:27 STEP  360 H= 360:00 +CPU= 25.033
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000360
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_6.zip
Uploading the intermediate file: upload_file_6.zip
  11:02:41 STEP  361 H= 361:00 +CPU= 13.488
Uploading trickle at timestep: 1296000
  11:02:54 STEP  362 H= 362:00 +CPU= 13.818
  11:03:15 STEP  363 H= 363:00 +CPU= 20.901
  11:03:29 STEP  364 H= 364:00 +CPU= 13.817
  11:03:43 STEP  365 H= 365:00 +CPU= 13.473
  11:04:05 STEP  366 H= 366:00 +CPU= 21.569
  11:04:19 STEP  367 H= 367:00 +CPU= 13.604
  11:04:33 STEP  368 H= 368:00 +CPU= 14.083
  11:04:55 STEP  369 H= 369:00 +CPU= 21.388
  11:05:10 STEP  370 H= 370:00 +CPU= 14.032
  11:05:24 STEP  371 H= 371:00 +CPU= 13.650
  11:05:48 STEP  372 H= 372:00 +CPU= 23.534
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000372
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000372
  11:06:02 STEP  373 H= 373:00 +CPU= 13.773
  11:06:17 STEP  374 H= 374:00 +CPU= 14.185
  11:06:40 STEP  375 H= 375:00 +CPU= 21.254
  11:06:54 STEP  376 H= 376:00 +CPU= 13.927
  11:07:07 STEP  377 H= 377:00 +CPU= 13.507
  11:07:29 STEP  378 H= 378:00 +CPU= 21.461
  11:07:43 STEP  379 H= 379:00 +CPU= 13.527
  11:07:57 STEP  380 H= 380:00 +CPU= 13.876
  11:08:18 STEP  381 H= 381:00 +CPU= 21.035
  11:08:32 STEP  382 H= 382:00 +CPU= 13.900
  11:08:47 STEP  383 H= 383:00 +CPU= 13.983
  11:09:12 STEP  384 H= 384:00 +CPU= 25.063
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000384
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_7.zip
Uploading the intermediate file: upload_file_7.zip
  11:09:26 STEP  385 H= 385:00 +CPU= 13.530
Uploading trickle at timestep: 1382400
  11:09:40 STEP  386 H= 386:00 +CPU= 13.905
  11:10:01 STEP  387 H= 387:00 +CPU= 21.051
  11:10:15 STEP  388 H= 388:00 +CPU= 13.882
  11:10:29 STEP  389 H= 389:00 +CPU= 13.510
  11:10:51 STEP  390 H= 390:00 +CPU= 21.467
  11:11:04 STEP  391 H= 391:00 +CPU= 13.495
  11:11:19 STEP  392 H= 392:00 +CPU= 13.902
  11:11:40 STEP  393 H= 393:00 +CPU= 21.092
  11:11:54 STEP  394 H= 394:00 +CPU= 13.876
  11:12:08 STEP  395 H= 395:00 +CPU= 13.530
  11:12:31 STEP  396 H= 396:00 +CPU= 23.254
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000396
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000396
  11:12:45 STEP  397 H= 397:00 +CPU= 13.592
  11:12:59 STEP  398 H= 398:00 +CPU= 13.921
  11:13:20 STEP  399 H= 399:00 +CPU= 20.990
  11:13:34 STEP  400 H= 400:00 +CPU= 13.864
  11:13:48 STEP  401 H= 401:00 +CPU= 13.515
  11:14:10 STEP  402 H= 402:00 +CPU= 21.459
  11:14:23 STEP  403 H= 403:00 +CPU= 13.514
  11:14:37 STEP  404 H= 404:00 +CPU= 13.860
  11:14:59 STEP  405 H= 405:00 +CPU= 21.004
  11:15:13 STEP  406 H= 406:00 +CPU= 13.848
  11:15:27 STEP  407 H= 407:00 +CPU= 13.959
  11:15:52 STEP  408 H= 408:00 +CPU= 25.038
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000408
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000408
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_8.zip
Uploading the intermediate file: upload_file_8.zip
  11:16:06 STEP  409 H= 409:00 +CPU= 13.537
Uploading trickle at timestep: 1468800
  11:16:20 STEP  410 H= 410:00 +CPU= 13.872
  11:16:42 STEP  411 H= 411:00 +CPU= 20.986
  11:16:56 STEP  412 H= 412:00 +CPU= 13.869
  11:17:09 STEP  413 H= 413:00 +CPU= 13.460
  11:17:31 STEP  414 H= 414:00 +CPU= 21.429
  11:17:44 STEP  415 H= 415:00 +CPU= 13.541
  11:17:58 STEP  416 H= 416:00 +CPU= 13.930
  11:18:20 STEP  417 H= 417:00 +CPU= 21.055
  11:18:34 STEP  418 H= 418:00 +CPU= 13.860
  11:18:48 STEP  419 H= 419:00 +CPU= 13.474
  11:19:11 STEP  420 H= 420:00 +CPU= 23.201
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000420
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000420
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000420
  11:19:25 STEP  421 H= 421:00 +CPU= 13.484
  11:19:39 STEP  422 H= 422:00 +CPU= 13.851
  11:20:00 STEP  423 H= 423:00 +CPU= 20.975
  11:20:14 STEP  424 H= 424:00 +CPU= 13.828
  11:20:28 STEP  425 H= 425:00 +CPU= 13.467
  11:20:49 STEP  426 H= 426:00 +CPU= 21.338
  11:21:03 STEP  427 H= 427:00 +CPU= 13.404
  11:21:17 STEP  428 H= 428:00 +CPU= 13.844
  11:21:38 STEP  429 H= 429:00 +CPU= 20.934
  11:21:52 STEP  430 H= 430:00 +CPU= 13.822
  11:22:06 STEP  431 H= 431:00 +CPU= 13.896
  11:22:31 STEP  432 H= 432:00 +CPU= 24.986
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000432
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000432
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_9.zip
Uploading the intermediate file: upload_file_9.zip
  11:22:45 STEP  433 H= 433:00 +CPU= 13.487
Uploading trickle at timestep: 1555200
  11:22:59 STEP  434 H= 434:00 +CPU= 13.797
  11:23:20 STEP  435 H= 435:00 +CPU= 20.989
  11:23:34 STEP  436 H= 436:00 +CPU= 13.853
  11:23:48 STEP  437 H= 437:00 +CPU= 13.444
  11:24:10 STEP  438 H= 438:00 +CPU= 21.379
  11:24:24 STEP  439 H= 439:00 +CPU= 13.501
  11:24:38 STEP  440 H= 440:00 +CPU= 13.858
  11:24:59 STEP  441 H= 441:00 +CPU= 20.898
  11:25:13 STEP  442 H= 442:00 +CPU= 13.866
  11:25:26 STEP  443 H= 443:00 +CPU= 13.481
  11:25:50 STEP  444 H= 444:00 +CPU= 23.103
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000444
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000444
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000444
  11:26:03 STEP  445 H= 445:00 +CPU= 13.477
  11:26:17 STEP  446 H= 446:00 +CPU= 13.806
  11:26:38 STEP  447 H= 447:00 +CPU= 20.951
  11:26:52 STEP  448 H= 448:00 +CPU= 13.792
  11:27:06 STEP  449 H= 449:00 +CPU= 13.452
  11:27:28 STEP  450 H= 450:00 +CPU= 21.387
  11:27:41 STEP  451 H= 451:00 +CPU= 13.468
  11:27:55 STEP  452 H= 452:00 +CPU= 13.855
  11:28:16 STEP  453 H= 453:00 +CPU= 20.947
  11:28:30 STEP  454 H= 454:00 +CPU= 13.815
  11:28:44 STEP  455 H= 455:00 +CPU= 13.831
  11:29:10 STEP  456 H= 456:00 +CPU= 25.007
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000456
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000456
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_10.zip
Uploading the intermediate file: upload_file_10.zip
  11:29:25 STEP  457 H= 457:00 +CPU= 13.711
Uploading trickle at timestep: 1641600
  11:29:39 STEP  458 H= 458:00 +CPU= 13.824
  11:30:00 STEP  459 H= 459:00 +CPU= 21.024
  11:30:14 STEP  460 H= 460:00 +CPU= 13.824
  11:30:28 STEP  461 H= 461:00 +CPU= 13.558
  11:30:51 STEP  462 H= 462:00 +CPU= 21.690
  11:31:05 STEP  463 H= 463:00 +CPU= 13.548
  11:31:19 STEP  464 H= 464:00 +CPU= 13.886
  11:31:41 STEP  465 H= 465:00 +CPU= 21.183
  11:31:56 STEP  466 H= 466:00 +CPU= 14.116
  11:32:10 STEP  467 H= 467:00 +CPU= 13.482
  11:32:33 STEP  468 H= 468:00 +CPU= 22.859
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000468
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000468
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000468
  11:32:47 STEP  469 H= 469:00 +CPU= 13.309
  11:33:01 STEP  470 H= 470:00 +CPU= 13.683
  11:33:22 STEP  471 H= 471:00 +CPU= 20.770
  11:33:35 STEP  472 H= 472:00 +CPU= 13.625
  11:33:49 STEP  473 H= 473:00 +CPU= 13.268
  11:34:12 STEP  474 H= 474:00 +CPU= 21.386
  11:34:26 STEP  475 H= 475:00 +CPU= 13.541
  11:34:40 STEP  476 H= 476:00 +CPU= 13.772
  11:35:02 STEP  477 H= 477:00 +CPU= 20.903
  11:35:16 STEP  478 H= 478:00 +CPU= 13.696
  11:35:30 STEP  479 H= 479:00 +CPU= 13.671
  11:35:55 STEP  480 H= 480:00 +CPU= 24.751
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000480
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000480
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_11.zip
Uploading the intermediate file: upload_file_11.zip
  11:36:09 STEP  481 H= 481:00 +CPU= 13.370
  11:36:23 STEP  482 H= 482:00 +CPU= 13.666
Uploading trickle at timestep: 1728000
  11:36:44 STEP  483 H= 483:00 +CPU= 20.837
  11:36:58 STEP  484 H= 484:00 +CPU= 13.683
  11:37:11 STEP  485 H= 485:00 +CPU= 13.322
  11:37:33 STEP  486 H= 486:00 +CPU= 21.178
  11:37:46 STEP  487 H= 487:00 +CPU= 13.276
  11:38:00 STEP  488 H= 488:00 +CPU= 13.632
  11:38:21 STEP  489 H= 489:00 +CPU= 20.788
  11:38:35 STEP  490 H= 490:00 +CPU= 13.669
  11:38:48 STEP  491 H= 491:00 +CPU= 13.157
  11:39:12 STEP  492 H= 492:00 +CPU= 22.767
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000492
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000492
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000492
  11:39:25 STEP  493 H= 493:00 +CPU= 13.221
  11:39:39 STEP  494 H= 494:00 +CPU= 13.605
  11:40:00 STEP  495 H= 495:00 +CPU= 20.832
  11:40:14 STEP  496 H= 496:00 +CPU= 13.790
  11:40:28 STEP  497 H= 497:00 +CPU= 13.347
  11:40:49 STEP  498 H= 498:00 +CPU= 21.134
  11:41:03 STEP  499 H= 499:00 +CPU= 13.255
  11:41:18 STEP  500 H= 500:00 +CPU= 13.856
  11:41:40 STEP  501 H= 501:00 +CPU= 21.024
  11:41:54 STEP  502 H= 502:00 +CPU= 13.771
  11:42:09 STEP  503 H= 503:00 +CPU= 13.803
  11:42:35 STEP  504 H= 504:00 +CPU= 24.897
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000504
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000504
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_12.zip
Uploading the intermediate file: upload_file_12.zip
  11:42:49 STEP  505 H= 505:00 +CPU= 13.434
  11:43:03 STEP  506 H= 506:00 +CPU= 13.718
Uploading trickle at timestep: 1814400
  11:43:25 STEP  507 H= 507:00 +CPU= 20.914
  11:43:39 STEP  508 H= 508:00 +CPU= 13.766
  11:43:53 STEP  509 H= 509:00 +CPU= 13.478
  11:44:15 STEP  510 H= 510:00 +CPU= 21.346
  11:44:29 STEP  511 H= 511:00 +CPU= 13.387
  11:44:43 STEP  512 H= 512:00 +CPU= 13.758
  11:45:04 STEP  513 H= 513:00 +CPU= 20.920
  11:45:19 STEP  514 H= 514:00 +CPU= 13.782
  11:45:33 STEP  515 H= 515:00 +CPU= 13.478
  11:45:56 STEP  516 H= 516:00 +CPU= 22.927
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000516
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000516
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000516
  11:46:10 STEP  517 H= 517:00 +CPU= 13.311
  11:46:23 STEP  518 H= 518:00 +CPU= 13.672
  11:46:45 STEP  519 H= 519:00 +CPU= 20.743
  11:46:58 STEP  520 H= 520:00 +CPU= 13.604
  11:47:12 STEP  521 H= 521:00 +CPU= 13.267
  11:47:33 STEP  522 H= 522:00 +CPU= 21.229
  11:47:47 STEP  523 H= 523:00 +CPU= 13.293
  11:48:01 STEP  524 H= 524:00 +CPU= 13.632
  11:48:22 STEP  525 H= 525:00 +CPU= 20.774
  11:48:36 STEP  526 H= 526:00 +CPU= 13.618
  11:48:49 STEP  527 H= 527:00 +CPU= 13.662
  11:49:14 STEP  528 H= 528:00 +CPU= 24.663
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000528
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000528
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_13.zip
Uploading the intermediate file: upload_file_13.zip
  11:49:28 STEP  529 H= 529:00 +CPU= 13.216
Uploading trickle at timestep: 1900800
  11:49:42 STEP  530 H= 530:00 +CPU= 13.619
  11:50:02 STEP  531 H= 531:00 +CPU= 20.654
  11:50:16 STEP  532 H= 532:00 +CPU= 13.574
  11:50:30 STEP  533 H= 533:00 +CPU= 13.237
  11:50:51 STEP  534 H= 534:00 +CPU= 21.158
  11:51:05 STEP  535 H= 535:00 +CPU= 13.310
  11:51:18 STEP  536 H= 536:00 +CPU= 13.615
  11:51:39 STEP  537 H= 537:00 +CPU= 20.729
  11:51:53 STEP  538 H= 538:00 +CPU= 13.634
  11:52:07 STEP  539 H= 539:00 +CPU= 13.250
  11:52:30 STEP  540 H= 540:00 +CPU= 22.914
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000540
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000540
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000540
  11:52:43 STEP  541 H= 541:00 +CPU= 13.292
  11:52:57 STEP  542 H= 542:00 +CPU= 13.695
  11:53:18 STEP  543 H= 543:00 +CPU= 20.776
  11:53:32 STEP  544 H= 544:00 +CPU= 13.626
  11:53:46 STEP  545 H= 545:00 +CPU= 13.263
  11:54:07 STEP  546 H= 546:00 +CPU= 21.280
  11:54:21 STEP  547 H= 547:00 +CPU= 13.402
  11:54:36 STEP  548 H= 548:00 +CPU= 13.738
  11:54:57 STEP  549 H= 549:00 +CPU= 20.912
  11:55:11 STEP  550 H= 550:00 +CPU= 13.734
  11:55:26 STEP  551 H= 551:00 +CPU= 13.838
  11:55:51 STEP  552 H= 552:00 +CPU= 25.011
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000552
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000552
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_14.zip
Uploading the intermediate file: upload_file_14.zip
  11:56:05 STEP  553 H= 553:00 +CPU= 13.470
  11:56:20 STEP  554 H= 554:00 +CPU= 13.807
Uploading trickle at timestep: 1987200
  11:56:41 STEP  555 H= 555:00 +CPU= 20.908
  11:56:55 STEP  556 H= 556:00 +CPU= 13.711
  11:57:09 STEP  557 H= 557:00 +CPU= 13.374
  11:57:30 STEP  558 H= 558:00 +CPU= 21.328
  11:57:44 STEP  559 H= 559:00 +CPU= 13.311
  11:57:58 STEP  560 H= 560:00 +CPU= 13.736
  11:58:19 STEP  561 H= 561:00 +CPU= 20.852
  11:58:33 STEP  562 H= 562:00 +CPU= 13.738
  11:58:46 STEP  563 H= 563:00 +CPU= 13.306
  11:59:09 STEP  564 H= 564:00 +CPU= 22.974
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000564
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000564
  11:59:23 STEP  565 H= 565:00 +CPU= 13.319
  11:59:37 STEP  566 H= 566:00 +CPU= 13.730
  11:59:58 STEP  567 H= 567:00 +CPU= 20.845
  12:00:14 STEP  568 H= 568:00 +CPU= 13.756
  12:00:30 STEP  569 H= 569:00 +CPU= 13.632
  12:00:56 STEP  570 H= 570:00 +CPU= 21.496
  12:01:12 STEP  571 H= 571:00 +CPU= 13.448
  12:01:29 STEP  572 H= 572:00 +CPU= 13.870
  12:01:55 STEP  573 H= 573:00 +CPU= 21.051
  12:02:12 STEP  574 H= 574:00 +CPU= 13.885
  12:02:29 STEP  575 H= 575:00 +CPU= 13.883
  12:02:59 STEP  576 H= 576:00 +CPU= 25.083
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000576
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000576
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_15.zip
Uploading the intermediate file: upload_file_15.zip
  12:03:15 STEP  577 H= 577:00 +CPU= 13.473
Uploading trickle at timestep: 2073600
  12:03:32 STEP  578 H= 578:00 +CPU= 13.940
  12:03:58 STEP  579 H= 579:00 +CPU= 21.146
  12:04:15 STEP  580 H= 580:00 +CPU= 13.938
  12:04:31 STEP  581 H= 581:00 +CPU= 13.565
  12:04:57 STEP  582 H= 582:00 +CPU= 21.611
  12:05:13 STEP  583 H= 583:00 +CPU= 13.586
  12:05:30 STEP  584 H= 584:00 +CPU= 13.968
  12:05:56 STEP  585 H= 585:00 +CPU= 21.197
  12:06:13 STEP  586 H= 586:00 +CPU= 13.949
  12:06:30 STEP  587 H= 587:00 +CPU= 13.540
  12:06:59 STEP  588 H= 588:00 +CPU= 23.323
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000588
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000588
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000588
  12:07:16 STEP  589 H= 589:00 +CPU= 13.734
  12:07:33 STEP  590 H= 590:00 +CPU= 14.154
  12:08:01 STEP  591 H= 591:00 +CPU= 21.421
  12:08:18 STEP  592 H= 592:00 +CPU= 14.147
  12:08:35 STEP  593 H= 593:00 +CPU= 13.673
  12:09:02 STEP  594 H= 594:00 +CPU= 21.759
  12:09:20 STEP  595 H= 595:00 +CPU= 13.732
  12:09:37 STEP  596 H= 596:00 +CPU= 14.025
  12:10:03 STEP  597 H= 597:00 +CPU= 21.245
  12:10:20 STEP  598 H= 598:00 +CPU= 13.987
  12:10:37 STEP  599 H= 599:00 +CPU= 13.818
  12:11:08 STEP  600 H= 600:00 +CPU= 25.297
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000600
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000600
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_16.zip
Uploading the intermediate file: upload_file_16.zip
  12:11:24 STEP  601 H= 601:00 +CPU= 13.670
Uploading trickle at timestep: 2160000
  12:11:41 STEP  602 H= 602:00 +CPU= 14.036
  12:12:08 STEP  603 H= 603:00 +CPU= 21.346
  12:12:26 STEP  604 H= 604:00 +CPU= 14.107
  12:12:44 STEP  605 H= 605:00 +CPU= 13.655
  12:13:11 STEP  606 H= 606:00 +CPU= 21.582
  12:13:28 STEP  607 H= 607:00 +CPU= 13.600
  12:13:45 STEP  608 H= 608:00 +CPU= 14.018
  12:14:11 STEP  609 H= 609:00 +CPU= 21.214
  12:14:29 STEP  610 H= 610:00 +CPU= 14.105
  12:14:46 STEP  611 H= 611:00 +CPU= 13.640
  12:15:15 STEP  612 H= 612:00 +CPU= 23.252
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000612
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000612
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000612
  12:15:31 STEP  613 H= 613:00 +CPU= 13.506
  12:15:48 STEP  614 H= 614:00 +CPU= 13.893
  12:16:15 STEP  615 H= 615:00 +CPU= 21.079
  12:16:32 STEP  616 H= 616:00 +CPU= 13.849
  12:16:49 STEP  617 H= 617:00 +CPU= 13.506
  12:17:15 STEP  618 H= 618:00 +CPU= 21.471
  12:17:32 STEP  619 H= 619:00 +CPU= 13.502
  12:17:50 STEP  620 H= 620:00 +CPU= 13.844
  12:18:16 STEP  621 H= 621:00 +CPU= 21.020
  12:18:33 STEP  622 H= 622:00 +CPU= 13.837
  12:18:50 STEP  623 H= 623:00 +CPU= 13.915
  12:19:18 STEP  624 H= 624:00 +CPU= 24.786
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+000624
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+000624
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMGGhpi1+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMSHhpi1+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12164643/ICMUAhpi1+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_1554_2021050100_123_946_12164643_1_r92312708_17.zip
Uploading the intermediate file: upload_file_17.zip
  12:19:31 STEP  625 H= 625:00 +CPU= 13.174
  12:19:45 STEP  626 H= 626:00 +CPU= 13.522
Uploading trickle at timestep: 2246400
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 2,246,400 6,714 0.0030
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 2,160,000 6,314 0.0029
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 2,073,600 5,913 0.0029
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,987,200 5,526 0.0028
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,900,800 5,131 0.0027
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,814,400 4,748 0.0026
09 May 2023 15:21:41 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,728,000 4,355 0.0025
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,641,600 3,956 0.0024
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,555,200 3,568 0.0023
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,468,800 3,170 0.0022
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,382,400 2,773 0.0020
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,296,000 2,379 0.0018
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,209,600 1,989 0.0016
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,123,200 1,594 0.0014
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 1,036,800 1,198 0.0012
09 May 2023 15:21:24 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 950,400 808 0.0009
09 May 2023 15:21:09 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 864,000 411 0.0005
09 May 2023 15:17:35 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 777,600 3,378 0.0043
09 May 2023 15:17:35 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 691,200 3,000 0.0043
09 May 2023 15:17:35 1527784 22248712 oifs_43r3_ps_1554_2021050100_123_946_12164643_1 604,800 2,619 0.0043


©2024 climateprediction.net