climateprediction.net home page
Task 22251033

Task 22251033

Name oifs_43r3_ps_0178_1981050100_123_950_12166822_0
Workunit 12166822
Created 20 Dec 2022, 18:50:29 UTC
Sent 20 Dec 2022, 18:55:36 UTC
Report deadline 19 Jan 2023, 18:55:36 UTC
Received 22 Dec 2022, 10:28:55 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1507812
Run time 1 days 1 hours 13 min 33 sec
CPU time 23 hours 10 min 28 sec
Validate state Invalid
Credit 13,547.52
Device peak FLOPS 5.42 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,769.77 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,827.65 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
73 H=2273:00 +CPU= 21.121
  05:57:38 STEP 2274 H=2274:00 +CPU= 33.570
  05:58:00 STEP 2275 H=2275:00 +CPU= 20.818
  05:58:22 STEP 2276 H=2276:00 +CPU= 21.486
  05:58:55 STEP 2277 H=2277:00 +CPU= 31.741
  05:59:18 STEP 2278 H=2278:00 +CPU= 21.883
  05:59:43 STEP 2279 H=2279:00 +CPU= 23.307
  06:00:26 STEP 2280 H=2280:00 +CPU= 39.668
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_94.zip
Uploading the intermediate file: upload_file_94.zip
  06:00:50 STEP 2281 H=2281:00 +CPU= 22.055
  06:01:15 STEP 2282 H=2282:00 +CPU= 22.904
  06:01:51 STEP 2283 H=2283:00 +CPU= 34.352
  06:02:16 STEP 2284 H=2284:00 +CPU= 23.174
  06:02:41 STEP 2285 H=2285:00 +CPU= 22.837
  06:03:20 STEP 2286 H=2286:00 +CPU= 36.360
  06:03:45 STEP 2287 H=2287:00 +CPU= 22.783
  06:04:12 STEP 2288 H=2288:00 +CPU= 24.259
  06:04:49 STEP 2289 H=2289:00 +CPU= 35.293
  06:05:15 STEP 2290 H=2290:00 +CPU= 23.578
  06:05:40 STEP 2291 H=2291:00 +CPU= 22.940
  06:06:23 STEP 2292 H=2292:00 +CPU= 38.625
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002292
  06:06:48 STEP 2293 H=2293:00 +CPU= 23.167
  06:07:13 STEP 2294 H=2294:00 +CPU= 22.977
  06:07:51 STEP 2295 H=2295:00 +CPU= 33.952
  06:08:17 STEP 2296 H=2296:00 +CPU= 22.622
  06:08:41 STEP 2297 H=2297:00 +CPU= 21.970
  06:09:19 STEP 2298 H=2298:00 +CPU= 34.986
  06:09:42 STEP 2299 H=2299:00 +CPU= 21.650
  06:10:06 STEP 2300 H=2300:00 +CPU= 22.280
  06:10:42 STEP 2301 H=2301:00 +CPU= 34.194
  06:11:06 STEP 2302 H=2302:00 +CPU= 22.919
  06:11:31 STEP 2303 H=2303:00 +CPU= 23.552
  06:12:13 STEP 2304 H=2304:00 +CPU= 39.235
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_95.zip
Uploading the intermediate file: upload_file_95.zip
  06:12:36 STEP 2305 H=2305:00 +CPU= 21.385
  06:13:00 STEP 2306 H=2306:00 +CPU= 22.820
  06:13:36 STEP 2307 H=2307:00 +CPU= 33.958
  06:14:00 STEP 2308 H=2308:00 +CPU= 22.956
  06:14:25 STEP 2309 H=2309:00 +CPU= 22.675
  06:15:04 STEP 2310 H=2310:00 +CPU= 35.145
  06:15:28 STEP 2311 H=2311:00 +CPU= 22.283
  06:15:52 STEP 2312 H=2312:00 +CPU= 22.907
  06:16:30 STEP 2313 H=2313:00 +CPU= 34.888
  06:16:54 STEP 2314 H=2314:00 +CPU= 22.803
  06:17:19 STEP 2315 H=2315:00 +CPU= 22.589
  06:18:00 STEP 2316 H=2316:00 +CPU= 38.595
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002316
  06:18:24 STEP 2317 H=2317:00 +CPU= 22.837
  06:18:49 STEP 2318 H=2318:00 +CPU= 23.144
  06:19:27 STEP 2319 H=2319:00 +CPU= 34.297
  06:19:52 STEP 2320 H=2320:00 +CPU= 22.981
  06:20:15 STEP 2321 H=2321:00 +CPU= 21.682
  06:20:54 STEP 2322 H=2322:00 +CPU= 35.561
  06:21:22 STEP 2323 H=2323:00 +CPU= 23.297
  06:21:46 STEP 2324 H=2324:00 +CPU= 22.505
  06:22:23 STEP 2325 H=2325:00 +CPU= 34.183
  06:22:48 STEP 2326 H=2326:00 +CPU= 23.073
  06:23:14 STEP 2327 H=2327:00 +CPU= 23.492
  06:23:58 STEP 2328 H=2328:00 +CPU= 40.133
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_96.zip
Uploading the intermediate file: upload_file_96.zip
  06:24:22 STEP 2329 H=2329:00 +CPU= 22.415
  06:24:47 STEP 2330 H=2330:00 +CPU= 22.797
  06:25:22 STEP 2331 H=2331:00 +CPU= 33.699
  06:25:47 STEP 2332 H=2332:00 +CPU= 22.772
  06:26:10 STEP 2333 H=2333:00 +CPU= 21.730
  06:26:47 STEP 2334 H=2334:00 +CPU= 35.038
  06:27:10 STEP 2335 H=2335:00 +CPU= 21.675
  06:27:36 STEP 2336 H=2336:00 +CPU= 23.174
  06:28:15 STEP 2337 H=2337:00 +CPU= 33.561
  06:28:40 STEP 2338 H=2338:00 +CPU= 22.560
  06:29:04 STEP 2339 H=2339:00 +CPU= 22.120
  06:29:48 STEP 2340 H=2340:00 +CPU= 38.967
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002340
  06:30:13 STEP 2341 H=2341:00 +CPU= 22.599
  06:30:38 STEP 2342 H=2342:00 +CPU= 23.192
  06:31:15 STEP 2343 H=2343:00 +CPU= 34.340
  06:31:41 STEP 2344 H=2344:00 +CPU= 23.752
  06:32:07 STEP 2345 H=2345:00 +CPU= 22.420
  06:32:44 STEP 2346 H=2346:00 +CPU= 35.300
  06:33:08 STEP 2347 H=2347:00 +CPU= 22.436
  06:33:33 STEP 2348 H=2348:00 +CPU= 23.102
  06:34:11 STEP 2349 H=2349:00 +CPU= 35.106
  06:34:37 STEP 2350 H=2350:00 +CPU= 23.357
  06:35:06 STEP 2351 H=2351:00 +CPU= 24.399
  06:35:50 STEP 2352 H=2352:00 +CPU= 40.188
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_97.zip
Uploading the intermediate file: upload_file_97.zip
  06:36:13 STEP 2353 H=2353:00 +CPU= 21.731
  06:36:36 STEP 2354 H=2354:00 +CPU= 22.461
  06:37:13 STEP 2355 H=2355:00 +CPU= 34.725
  06:37:39 STEP 2356 H=2356:00 +CPU= 22.768
  06:38:04 STEP 2357 H=2357:00 +CPU= 22.060
  06:38:43 STEP 2358 H=2358:00 +CPU= 35.251
  06:39:07 STEP 2359 H=2359:00 +CPU= 21.926
  06:39:31 STEP 2360 H=2360:00 +CPU= 22.333
  06:40:08 STEP 2361 H=2361:00 +CPU= 33.755
  06:40:34 STEP 2362 H=2362:00 +CPU= 24.023
  06:40:59 STEP 2363 H=2363:00 +CPU= 22.187
  06:41:40 STEP 2364 H=2364:00 +CPU= 38.766
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002364
  06:42:04 STEP 2365 H=2365:00 +CPU= 22.576
  06:42:30 STEP 2366 H=2366:00 +CPU= 22.559
  06:43:05 STEP 2367 H=2367:00 +CPU= 33.122
  06:43:29 STEP 2368 H=2368:00 +CPU= 22.248
  06:43:55 STEP 2369 H=2369:00 +CPU= 22.041
  06:44:34 STEP 2370 H=2370:00 +CPU= 36.020
  06:44:58 STEP 2371 H=2371:00 +CPU= 22.529
  06:45:23 STEP 2372 H=2372:00 +CPU= 23.549
  06:46:00 STEP 2373 H=2373:00 +CPU= 35.057
  06:46:25 STEP 2374 H=2374:00 +CPU= 23.642
  06:46:50 STEP 2375 H=2375:00 +CPU= 23.545
  06:47:35 STEP 2376 H=2376:00 +CPU= 40.929
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_98.zip
Uploading the intermediate file: upload_file_98.zip
  06:47:59 STEP 2377 H=2377:00 +CPU= 22.731
  06:48:23 STEP 2378 H=2378:00 +CPU= 22.920
  06:49:03 STEP 2379 H=2379:00 +CPU= 35.357
  06:49:29 STEP 2380 H=2380:00 +CPU= 24.080
  06:49:53 STEP 2381 H=2381:00 +CPU= 23.147
  06:50:31 STEP 2382 H=2382:00 +CPU= 35.725
  06:50:58 STEP 2383 H=2383:00 +CPU= 23.140
  06:51:24 STEP 2384 H=2384:00 +CPU= 23.078
  06:52:01 STEP 2385 H=2385:00 +CPU= 34.830
  06:52:26 STEP 2386 H=2386:00 +CPU= 23.466
  06:52:50 STEP 2387 H=2387:00 +CPU= 22.235
  06:53:32 STEP 2388 H=2388:00 +CPU= 39.251
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002388
  06:53:55 STEP 2389 H=2389:00 +CPU= 21.771
  06:54:20 STEP 2390 H=2390:00 +CPU= 23.383
  06:54:57 STEP 2391 H=2391:00 +CPU= 34.916
  06:55:24 STEP 2392 H=2392:00 +CPU= 23.792
  06:55:51 STEP 2393 H=2393:00 +CPU= 23.807
  06:56:30 STEP 2394 H=2394:00 +CPU= 36.049
  06:56:55 STEP 2395 H=2395:00 +CPU= 21.456
  06:57:19 STEP 2396 H=2396:00 +CPU= 22.078
  06:57:55 STEP 2397 H=2397:00 +CPU= 34.308
  06:58:21 STEP 2398 H=2398:00 +CPU= 23.423
  06:58:46 STEP 2399 H=2399:00 +CPU= 23.776
  06:59:31 STEP 2400 H=2400:00 +CPU= 40.853
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_99.zip
Uploading the intermediate file: upload_file_99.zip
  06:59:55 STEP 2401 H=2401:00 +CPU= 22.788
  07:00:20 STEP 2402 H=2402:00 +CPU= 22.580
  07:00:58 STEP 2403 H=2403:00 +CPU= 35.443
  07:01:24 STEP 2404 H=2404:00 +CPU= 23.656
  07:01:47 STEP 2405 H=2405:00 +CPU= 22.091
  07:02:26 STEP 2406 H=2406:00 +CPU= 35.840
  07:02:53 STEP 2407 H=2407:00 +CPU= 23.299
  07:03:22 STEP 2408 H=2408:00 +CPU= 24.588
  07:04:02 STEP 2409 H=2409:00 +CPU= 35.564
  07:04:26 STEP 2410 H=2410:00 +CPU= 22.394
  07:04:49 STEP 2411 H=2411:00 +CPU= 21.640
  07:05:32 STEP 2412 H=2412:00 +CPU= 39.264
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002412
  07:05:56 STEP 2413 H=2413:00 +CPU= 22.303
  07:06:20 STEP 2414 H=2414:00 +CPU= 22.695
  07:06:58 STEP 2415 H=2415:00 +CPU= 34.914
  07:07:23 STEP 2416 H=2416:00 +CPU= 23.431
  07:07:48 STEP 2417 H=2417:00 +CPU= 22.394
  07:08:26 STEP 2418 H=2418:00 +CPU= 35.642
  07:08:51 STEP 2419 H=2419:00 +CPU= 23.951
  07:09:16 STEP 2420 H=2420:00 +CPU= 23.912
  07:09:54 STEP 2421 H=2421:00 +CPU= 35.059
  07:10:19 STEP 2422 H=2422:00 +CPU= 24.374
  07:10:46 STEP 2423 H=2423:00 +CPU= 23.763
  07:11:30 STEP 2424 H=2424:00 +CPU= 40.063
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_100.zip
Uploading the intermediate file: upload_file_100.zip
  07:11:53 STEP 2425 H=2425:00 +CPU= 22.024
  07:12:18 STEP 2426 H=2426:00 +CPU= 22.998
  07:12:55 STEP 2427 H=2427:00 +CPU= 34.643
  07:13:19 STEP 2428 H=2428:00 +CPU= 22.935
  07:13:43 STEP 2429 H=2429:00 +CPU= 22.373
  07:14:22 STEP 2430 H=2430:00 +CPU= 36.491
  07:14:47 STEP 2431 H=2431:00 +CPU= 23.455
  07:15:12 STEP 2432 H=2432:00 +CPU= 23.606
  07:15:50 STEP 2433 H=2433:00 +CPU= 35.091
  07:16:16 STEP 2434 H=2434:00 +CPU= 23.980
  07:16:40 STEP 2435 H=2435:00 +CPU= 23.248
  07:17:24 STEP 2436 H=2436:00 +CPU= 38.273
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002436
  07:17:50 STEP 2437 H=2437:00 +CPU= 22.412
  07:18:15 STEP 2438 H=2438:00 +CPU= 23.205
  07:18:53 STEP 2439 H=2439:00 +CPU= 34.549
  07:19:18 STEP 2440 H=2440:00 +CPU= 23.783
  07:19:43 STEP 2441 H=2441:00 +CPU= 22.692
  07:20:27 STEP 2442 H=2442:00 +CPU= 36.672
  07:20:53 STEP 2443 H=2443:00 +CPU= 23.096
  07:21:19 STEP 2444 H=2444:00 +CPU= 23.228
  07:21:57 STEP 2445 H=2445:00 +CPU= 34.320
  07:22:21 STEP 2446 H=2446:00 +CPU= 23.342
  07:22:47 STEP 2447 H=2447:00 +CPU= 22.975
  07:23:32 STEP 2448 H=2448:00 +CPU= 40.743
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_101.zip
Uploading the intermediate file: upload_file_101.zip
  07:23:55 STEP 2449 H=2449:00 +CPU= 22.077
Uploading trickle at timestep: 8812800
  07:24:19 STEP 2450 H=2450:00 +CPU= 22.435
  07:24:57 STEP 2451 H=2451:00 +CPU= 34.926
  07:25:23 STEP 2452 H=2452:00 +CPU= 23.460
  07:25:49 STEP 2453 H=2453:00 +CPU= 22.399
  07:26:29 STEP 2454 H=2454:00 +CPU= 36.072
  07:26:56 STEP 2455 H=2455:00 +CPU= 23.678
  07:27:22 STEP 2456 H=2456:00 +CPU= 23.679
  07:28:00 STEP 2457 H=2457:00 +CPU= 35.268
  07:28:24 STEP 2458 H=2458:00 +CPU= 23.615
  07:28:47 STEP 2459 H=2459:00 +CPU= 21.313
  07:29:27 STEP 2460 H=2460:00 +CPU= 38.305
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002460
  07:29:51 STEP 2461 H=2461:00 +CPU= 23.146
  07:30:15 STEP 2462 H=2462:00 +CPU= 22.873
  07:30:52 STEP 2463 H=2463:00 +CPU= 34.706
  07:31:17 STEP 2464 H=2464:00 +CPU= 23.352
  07:31:42 STEP 2465 H=2465:00 +CPU= 22.256
  07:32:20 STEP 2466 H=2466:00 +CPU= 35.463
  07:32:44 STEP 2467 H=2467:00 +CPU= 23.251
  07:33:09 STEP 2468 H=2468:00 +CPU= 22.928
  07:33:44 STEP 2469 H=2469:00 +CPU= 33.481
  07:34:08 STEP 2470 H=2470:00 +CPU= 22.365
  07:34:33 STEP 2471 H=2471:00 +CPU= 23.475
  07:35:18 STEP 2472 H=2472:00 +CPU= 40.400
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_102.zip
Uploading the intermediate file: upload_file_102.zip
  07:35:42 STEP 2473 H=2473:00 +CPU= 21.955
  07:36:06 STEP 2474 H=2474:00 +CPU= 23.108
  07:36:43 STEP 2475 H=2475:00 +CPU= 34.850
  07:37:08 STEP 2476 H=2476:00 +CPU= 22.976
  07:37:32 STEP 2477 H=2477:00 +CPU= 22.537
  07:38:10 STEP 2478 H=2478:00 +CPU= 35.586
  07:38:35 STEP 2479 H=2479:00 +CPU= 23.060
  07:39:00 STEP 2480 H=2480:00 +CPU= 23.369
  07:39:36 STEP 2481 H=2481:00 +CPU= 33.935
  07:40:02 STEP 2482 H=2482:00 +CPU= 23.876
  07:40:29 STEP 2483 H=2483:00 +CPU= 22.547
  07:41:11 STEP 2484 H=2484:00 +CPU= 38.516
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002484
  07:41:36 STEP 2485 H=2485:00 +CPU= 22.584
  07:42:00 STEP 2486 H=2486:00 +CPU= 22.643
  07:42:37 STEP 2487 H=2487:00 +CPU= 34.497
  07:43:01 STEP 2488 H=2488:00 +CPU= 23.114
  07:43:26 STEP 2489 H=2489:00 +CPU= 22.157
  07:44:04 STEP 2490 H=2490:00 +CPU= 35.662
  07:44:29 STEP 2491 H=2491:00 +CPU= 23.206
  07:44:53 STEP 2492 H=2492:00 +CPU= 23.028
  07:45:31 STEP 2493 H=2493:00 +CPU= 35.022
  07:45:56 STEP 2494 H=2494:00 +CPU= 23.624
  07:46:22 STEP 2495 H=2495:00 +CPU= 23.192
  07:47:06 STEP 2496 H=2496:00 +CPU= 40.121
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_103.zip
Uploading the intermediate file: upload_file_103.zip
  07:47:29 STEP 2497 H=2497:00 +CPU= 21.684
  07:47:52 STEP 2498 H=2498:00 +CPU= 21.424
  07:48:26 STEP 2499 H=2499:00 +CPU= 32.592
  07:48:50 STEP 2500 H=2500:00 +CPU= 22.697
  07:49:14 STEP 2501 H=2501:00 +CPU= 22.572
  07:49:52 STEP 2502 H=2502:00 +CPU= 35.232
  07:50:17 STEP 2503 H=2503:00 +CPU= 23.475
  07:50:43 STEP 2504 H=2504:00 +CPU= 23.742
  07:51:19 STEP 2505 H=2505:00 +CPU= 34.329
  07:51:45 STEP 2506 H=2506:00 +CPU= 23.568
  07:52:08 STEP 2507 H=2507:00 +CPU= 22.164
  07:52:49 STEP 2508 H=2508:00 +CPU= 38.665
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002508
  07:53:13 STEP 2509 H=2509:00 +CPU= 22.429
  07:53:38 STEP 2510 H=2510:00 +CPU= 23.164
  07:54:14 STEP 2511 H=2511:00 +CPU= 34.807
  07:54:40 STEP 2512 H=2512:00 +CPU= 23.470
  07:55:04 STEP 2513 H=2513:00 +CPU= 22.841
  07:55:42 STEP 2514 H=2514:00 +CPU= 35.233
  07:56:06 STEP 2515 H=2515:00 +CPU= 22.534
  07:56:30 STEP 2516 H=2516:00 +CPU= 23.118
  07:57:09 STEP 2517 H=2517:00 +CPU= 35.049
  07:57:34 STEP 2518 H=2518:00 +CPU= 23.567
  07:57:58 STEP 2519 H=2519:00 +CPU= 22.990
  07:58:44 STEP 2520 H=2520:00 +CPU= 40.717
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_104.zip
Uploading the intermediate file: upload_file_104.zip
  07:59:09 STEP 2521 H=2521:00 +CPU= 22.747
  07:59:34 STEP 2522 H=2522:00 +CPU= 23.095
  08:00:10 STEP 2523 H=2523:00 +CPU= 34.718
  08:00:37 STEP 2524 H=2524:00 +CPU= 23.205
  08:01:03 STEP 2525 H=2525:00 +CPU= 23.744
  08:01:42 STEP 2526 H=2526:00 +CPU= 36.305
  08:02:06 STEP 2527 H=2527:00 +CPU= 21.867
  08:02:32 STEP 2528 H=2528:00 +CPU= 23.337
  08:03:11 STEP 2529 H=2529:00 +CPU= 34.644
  08:03:34 STEP 2530 H=2530:00 +CPU= 22.982
  08:03:58 STEP 2531 H=2531:00 +CPU= 22.638
  08:04:39 STEP 2532 H=2532:00 +CPU= 38.197
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002532
  08:05:03 STEP 2533 H=2533:00 +CPU= 21.670
  08:05:27 STEP 2534 H=2534:00 +CPU= 22.789
  08:06:05 STEP 2535 H=2535:00 +CPU= 34.525
  08:06:31 STEP 2536 H=2536:00 +CPU= 22.669
  08:06:55 STEP 2537 H=2537:00 +CPU= 22.684
  08:07:35 STEP 2538 H=2538:00 +CPU= 36.021
  08:08:00 STEP 2539 H=2539:00 +CPU= 22.753
  08:08:25 STEP 2540 H=2540:00 +CPU= 23.527
  08:09:02 STEP 2541 H=2541:00 +CPU= 34.948
  08:09:26 STEP 2542 H=2542:00 +CPU= 23.133
  08:09:50 STEP 2543 H=2543:00 +CPU= 23.104
  08:10:36 STEP 2544 H=2544:00 +CPU= 40.567
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_105.zip
Uploading the intermediate file: upload_file_105.zip
  08:10:59 STEP 2545 H=2545:00 +CPU= 21.892
  08:11:26 STEP 2546 H=2546:00 +CPU= 23.748
  08:12:04 STEP 2547 H=2547:00 +CPU= 35.315
  08:12:28 STEP 2548 H=2548:00 +CPU= 22.795
  08:12:54 STEP 2549 H=2549:00 +CPU= 23.138
  08:13:32 STEP 2550 H=2550:00 +CPU= 35.916
  08:13:58 STEP 2551 H=2551:00 +CPU= 22.589
  08:14:24 STEP 2552 H=2552:00 +CPU= 23.221
  08:15:01 STEP 2553 H=2553:00 +CPU= 34.594
  08:15:25 STEP 2554 H=2554:00 +CPU= 22.710
  08:15:49 STEP 2555 H=2555:00 +CPU= 22.588
  08:16:31 STEP 2556 H=2556:00 +CPU= 38.626
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002556
  08:16:56 STEP 2557 H=2557:00 +CPU= 22.729
  08:17:20 STEP 2558 H=2558:00 +CPU= 23.109
  08:17:58 STEP 2559 H=2559:00 +CPU= 34.726
  08:18:23 STEP 2560 H=2560:00 +CPU= 22.946
  08:18:48 STEP 2561 H=2561:00 +CPU= 23.027
  08:19:28 STEP 2562 H=2562:00 +CPU= 36.659
  08:19:52 STEP 2563 H=2563:00 +CPU= 22.462
  08:20:17 STEP 2564 H=2564:00 +CPU= 23.680
  08:20:55 STEP 2565 H=2565:00 +CPU= 35.065
  08:21:19 STEP 2566 H=2566:00 +CPU= 22.746
  08:21:45 STEP 2567 H=2567:00 +CPU= 24.181
  08:22:30 STEP 2568 H=2568:00 +CPU= 40.133
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_106.zip
Uploading the intermediate file: upload_file_106.zip
  08:22:55 STEP 2569 H=2569:00 +CPU= 22.235
  08:23:19 STEP 2570 H=2570:00 +CPU= 23.202
  08:23:58 STEP 2571 H=2571:00 +CPU= 35.176
  08:24:23 STEP 2572 H=2572:00 +CPU= 22.430
  08:24:45 STEP 2573 H=2573:00 +CPU= 21.280
  08:25:22 STEP 2574 H=2574:00 +CPU= 34.519
  08:25:48 STEP 2575 H=2575:00 +CPU= 22.539
  08:26:16 STEP 2576 H=2576:00 +CPU= 23.811
  08:26:56 STEP 2577 H=2577:00 +CPU= 35.515
  08:27:21 STEP 2578 H=2578:00 +CPU= 23.004
  08:27:45 STEP 2579 H=2579:00 +CPU= 22.530
  08:28:27 STEP 2580 H=2580:00 +CPU= 39.324
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002580
  08:28:50 STEP 2581 H=2581:00 +CPU= 21.782
  08:29:19 STEP 2582 H=2582:00 +CPU= 23.607
  08:29:56 STEP 2583 H=2583:00 +CPU= 35.363
  08:30:20 STEP 2584 H=2584:00 +CPU= 22.452
  08:30:44 STEP 2585 H=2585:00 +CPU= 22.858
  08:31:22 STEP 2586 H=2586:00 +CPU= 35.234
  08:31:47 STEP 2587 H=2587:00 +CPU= 22.679
  08:32:14 STEP 2588 H=2588:00 +CPU= 24.092
  08:37:55 STEP 2589 H=2589:00 +CPU= 34.832
  08:38:22 STEP 2590 H=2590:00 +CPU= 22.932
  08:46:40 STEP 2591 H=2591:00 +CPU= 22.720
  08:47:24 STEP 2592 H=2592:00 +CPU= 38.633
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_107.zip
Uploading the intermediate file: upload_file_107.zip
  08:47:48 STEP 2593 H=2593:00 +CPU= 22.233
  08:48:14 STEP 2594 H=2594:00 +CPU= 23.539
  08:49:05 STEP 2595 H=2595:00 +CPU= 34.959
  08:49:35 STEP 2596 H=2596:00 +CPU= 25.321
  08:49:58 STEP 2597 H=2597:00 +CPU= 21.143
  08:51:07 STEP 2598 H=2598:00 +CPU= 34.412
  08:52:17 STEP 2599 H=2599:00 +CPU= 21.851
  08:52:51 STEP 2600 H=2600:00 +CPU= 23.265
  08:53:29 STEP 2601 H=2601:00 +CPU= 34.756
  08:53:54 STEP 2602 H=2602:00 +CPU= 23.335
  08:54:18 STEP 2603 H=2603:00 +CPU= 22.880
  08:54:57 STEP 2604 H=2604:00 +CPU= 38.080
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002604
  08:55:22 STEP 2605 H=2605:00 +CPU= 22.274
  08:55:50 STEP 2606 H=2606:00 +CPU= 23.527
  08:56:26 STEP 2607 H=2607:00 +CPU= 34.613
  08:56:51 STEP 2608 H=2608:00 +CPU= 23.863
  08:57:16 STEP 2609 H=2609:00 +CPU= 22.249
  08:57:54 STEP 2610 H=2610:00 +CPU= 35.428
  08:58:19 STEP 2611 H=2611:00 +CPU= 23.193
  08:58:44 STEP 2612 H=2612:00 +CPU= 23.671
  08:59:22 STEP 2613 H=2613:00 +CPU= 34.792
  08:59:47 STEP 2614 H=2614:00 +CPU= 23.710
  09:00:12 STEP 2615 H=2615:00 +CPU= 23.289
  09:00:57 STEP 2616 H=2616:00 +CPU= 40.910
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_108.zip
Uploading the intermediate file: upload_file_108.zip
  09:01:22 STEP 2617 H=2617:00 +CPU= 21.965
  09:01:46 STEP 2618 H=2618:00 +CPU= 22.871
  09:02:24 STEP 2619 H=2619:00 +CPU= 35.040
  09:02:48 STEP 2620 H=2620:00 +CPU= 22.799
  09:03:13 STEP 2621 H=2621:00 +CPU= 22.244
  09:03:55 STEP 2622 H=2622:00 +CPU= 36.402
  09:04:18 STEP 2623 H=2623:00 +CPU= 22.060
  09:04:44 STEP 2624 H=2624:00 +CPU= 23.300
  09:05:23 STEP 2625 H=2625:00 +CPU= 35.085
  09:05:48 STEP 2626 H=2626:00 +CPU= 23.559
  09:06:13 STEP 2627 H=2627:00 +CPU= 22.441
  09:06:53 STEP 2628 H=2628:00 +CPU= 38.526
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002628
  09:07:18 STEP 2629 H=2629:00 +CPU= 23.349
  09:07:43 STEP 2630 H=2630:00 +CPU= 23.477
  09:08:19 STEP 2631 H=2631:00 +CPU= 34.381
  09:08:45 STEP 2632 H=2632:00 +CPU= 23.514
  09:09:07 STEP 2633 H=2633:00 +CPU= 21.169
  09:09:43 STEP 2634 H=2634:00 +CPU= 33.158
  09:10:06 STEP 2635 H=2635:00 +CPU= 21.947
  09:10:30 STEP 2636 H=2636:00 +CPU= 22.015
  09:11:08 STEP 2637 H=2637:00 +CPU= 34.001
  09:11:34 STEP 2638 H=2638:00 +CPU= 23.266
  09:12:02 STEP 2639 H=2639:00 +CPU= 22.586
  09:12:45 STEP 2640 H=2640:00 +CPU= 39.394
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_109.zip
Uploading the intermediate file: upload_file_109.zip
  09:13:10 STEP 2641 H=2641:00 +CPU= 22.072
  09:13:34 STEP 2642 H=2642:00 +CPU= 22.278
  09:14:11 STEP 2643 H=2643:00 +CPU= 34.448
  09:14:35 STEP 2644 H=2644:00 +CPU= 22.355
  09:14:58 STEP 2645 H=2645:00 +CPU= 22.050
  09:15:35 STEP 2646 H=2646:00 +CPU= 34.869
  09:15:59 STEP 2647 H=2647:00 +CPU= 21.957
  09:16:23 STEP 2648 H=2648:00 +CPU= 22.829
  09:16:59 STEP 2649 H=2649:00 +CPU= 33.876
  09:17:22 STEP 2650 H=2650:00 +CPU= 22.254
  09:17:46 STEP 2651 H=2651:00 +CPU= 22.098
  09:18:25 STEP 2652 H=2652:00 +CPU= 37.303
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002652
  09:18:48 STEP 2653 H=2653:00 +CPU= 21.720
  09:19:12 STEP 2654 H=2654:00 +CPU= 22.622
  09:19:48 STEP 2655 H=2655:00 +CPU= 34.320
  09:20:13 STEP 2656 H=2656:00 +CPU= 22.678
  09:20:36 STEP 2657 H=2657:00 +CPU= 22.056
  09:21:13 STEP 2658 H=2658:00 +CPU= 34.990
  09:21:37 STEP 2659 H=2659:00 +CPU= 21.891
  09:22:00 STEP 2660 H=2660:00 +CPU= 22.560
  09:22:37 STEP 2661 H=2661:00 +CPU= 34.294
  09:23:00 STEP 2662 H=2662:00 +CPU= 22.250
  09:23:25 STEP 2663 H=2663:00 +CPU= 22.549
  09:24:07 STEP 2664 H=2664:00 +CPU= 39.505
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_110.zip
Uploading the intermediate file: upload_file_110.zip
  09:24:31 STEP 2665 H=2665:00 +CPU= 22.149
  09:24:54 STEP 2666 H=2666:00 +CPU= 21.662
  09:25:29 STEP 2667 H=2667:00 +CPU= 33.994
  09:25:54 STEP 2668 H=2668:00 +CPU= 22.808
  09:26:16 STEP 2669 H=2669:00 +CPU= 21.174
  09:26:54 STEP 2670 H=2670:00 +CPU= 34.884
  09:27:20 STEP 2671 H=2671:00 +CPU= 23.069
  09:27:43 STEP 2672 H=2672:00 +CPU= 21.584
  09:28:19 STEP 2673 H=2673:00 +CPU= 34.130
  09:28:42 STEP 2674 H=2674:00 +CPU= 22.380
  09:29:04 STEP 2675 H=2675:00 +CPU= 20.832
  09:29:43 STEP 2676 H=2676:00 +CPU= 36.511
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002676
  09:30:06 STEP 2677 H=2677:00 +CPU= 21.336
  09:30:30 STEP 2678 H=2678:00 +CPU= 22.749
  09:31:07 STEP 2679 H=2679:00 +CPU= 34.541
  09:31:31 STEP 2680 H=2680:00 +CPU= 22.911
  09:31:56 STEP 2681 H=2681:00 +CPU= 22.450
  09:32:34 STEP 2682 H=2682:00 +CPU= 35.645
  09:32:58 STEP 2683 H=2683:00 +CPU= 22.508
  09:33:23 STEP 2684 H=2684:00 +CPU= 23.319
  09:34:02 STEP 2685 H=2685:00 +CPU= 34.569
  09:34:26 STEP 2686 H=2686:00 +CPU= 22.854
  09:34:49 STEP 2687 H=2687:00 +CPU= 22.690
  09:35:32 STEP 2688 H=2688:00 +CPU= 39.055
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_111.zip
Uploading the intermediate file: upload_file_111.zip
Uploading trickle at timestep: 9676800
  09:35:55 STEP 2689 H=2689:00 +CPU= 21.709
  09:36:18 STEP 2690 H=2690:00 +CPU= 22.191
  09:36:54 STEP 2691 H=2691:00 +CPU= 34.151
  09:37:19 STEP 2692 H=2692:00 +CPU= 23.145
  09:37:42 STEP 2693 H=2693:00 +CPU= 21.511
  09:38:20 STEP 2694 H=2694:00 +CPU= 35.342
  09:38:44 STEP 2695 H=2695:00 +CPU= 22.886
  09:39:08 STEP 2696 H=2696:00 +CPU= 22.726
  09:39:44 STEP 2697 H=2697:00 +CPU= 34.654
  09:40:11 STEP 2698 H=2698:00 +CPU= 24.962
  09:40:33 STEP 2699 H=2699:00 +CPU= 21.651
  09:41:13 STEP 2700 H=2700:00 +CPU= 38.146
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002700
  09:41:39 STEP 2701 H=2701:00 +CPU= 22.714
  09:42:02 STEP 2702 H=2702:00 +CPU= 22.324
  09:42:37 STEP 2703 H=2703:00 +CPU= 33.710
  09:43:05 STEP 2704 H=2704:00 +CPU= 23.927
  09:43:31 STEP 2705 H=2705:00 +CPU= 22.490
  09:44:08 STEP 2706 H=2706:00 +CPU= 35.151
  09:44:31 STEP 2707 H=2707:00 +CPU= 22.279
  09:44:56 STEP 2708 H=2708:00 +CPU= 23.342
  09:45:31 STEP 2709 H=2709:00 +CPU= 34.124
  09:45:56 STEP 2710 H=2710:00 +CPU= 22.898
  09:46:20 STEP 2711 H=2711:00 +CPU= 23.031
  09:47:04 STEP 2712 H=2712:00 +CPU= 40.566
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_112.zip
Uploading the intermediate file: upload_file_112.zip
  09:47:27 STEP 2713 H=2713:00 +CPU= 22.002
  09:47:51 STEP 2714 H=2714:00 +CPU= 23.111
  09:48:28 STEP 2715 H=2715:00 +CPU= 34.527
  09:48:52 STEP 2716 H=2716:00 +CPU= 23.087
  09:49:15 STEP 2717 H=2717:00 +CPU= 21.714
  09:49:53 STEP 2718 H=2718:00 +CPU= 35.566
  09:50:16 STEP 2719 H=2719:00 +CPU= 22.466
  09:50:41 STEP 2720 H=2720:00 +CPU= 22.711
  09:51:17 STEP 2721 H=2721:00 +CPU= 34.256
  09:51:41 STEP 2722 H=2722:00 +CPU= 23.163
  09:52:04 STEP 2723 H=2723:00 +CPU= 21.851
  09:52:45 STEP 2724 H=2724:00 +CPU= 38.186
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002724
  09:53:08 STEP 2725 H=2725:00 +CPU= 22.800
  09:53:33 STEP 2726 H=2726:00 +CPU= 22.825
  09:54:10 STEP 2727 H=2727:00 +CPU= 34.683
  09:54:35 STEP 2728 H=2728:00 +CPU= 23.081
  09:54:58 STEP 2729 H=2729:00 +CPU= 21.581
  09:55:37 STEP 2730 H=2730:00 +CPU= 35.282
  09:56:00 STEP 2731 H=2731:00 +CPU= 21.482
  09:56:23 STEP 2732 H=2732:00 +CPU= 21.449
  09:56:58 STEP 2733 H=2733:00 +CPU= 32.886
  09:57:22 STEP 2734 H=2734:00 +CPU= 23.112
  09:57:46 STEP 2735 H=2735:00 +CPU= 22.449
  09:58:28 STEP 2736 H=2736:00 +CPU= 39.141
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_113.zip
Uploading the intermediate file: upload_file_113.zip
  09:58:51 STEP 2737 H=2737:00 +CPU= 21.276
  09:59:14 STEP 2738 H=2738:00 +CPU= 21.912
  09:59:50 STEP 2739 H=2739:00 +CPU= 33.004
  10:00:15 STEP 2740 H=2740:00 +CPU= 22.945
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0178
(argv4) batchid: 950
(argv5) wuid: 12166822
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/9
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_eda5b301845cbe844fbf900b53270c9a to: /var/lib/boinc/slots/9/oifs_43r3_ps_0178_1981050100_123_950_12166822.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/9/oifs_43r3_ps_0178_1981050100_123_950_12166822.zip
ic_ancil_file: ic_ancil_12166822
ifsdata_file: ifsdata_12166822
climate_data_file: clim_data_12166822
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_437ff8b2e3c01312d73fa027abaa7e81 to: /var/lib/boinc/slots/9/ic_ancil_12166822.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/9/ic_ancil_12166822.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b536d640638811c1044382dd04c92df2 to: /var/lib/boinc/slots/9/ifsdata/ifsdata_12166822.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/9/ifsdata/ifsdata_12166822.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/9/159l_2/clim_data_12166822.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/9/159l_2/clim_data_12166822.zip
Checking for progress XML file: /var/lib/boinc/slots/9/progress_file_12166822.xml
Opened progress file ok : /var/lib/boinc/slots/9/progress_file_12166822.xml
-- Model is restarting --
Adjusting last_iter, 2741, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/9/progress_file_12166822.xml
last_cpu_time: 80974
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650
The child process has been launched with process id: 29693
Executing the command: /var/lib/boinc/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:zion.faskatech.net:1:1:29693:29693] [20221222:104016:1671702016:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:40:59 STEP 2736 H=2736:00 +CPU= 37.471
  10:41:23 STEP 2737 H=2737:00 +CPU= 21.848
  10:41:48 STEP 2738 H=2738:00 +CPU= 22.053
  10:42:26 STEP 2739 H=2739:00 +CPU= 33.121
  10:42:49 STEP 2740 H=2740:00 +CPU= 21.860
  10:43:12 STEP 2741 H=2741:00 +CPU= 21.513
  10:43:47 STEP 2742 H=2742:00 +CPU= 33.443
  10:44:10 STEP 2743 H=2743:00 +CPU= 21.001
  10:44:33 STEP 2744 H=2744:00 +CPU= 22.127
  10:45:08 STEP 2745 H=2745:00 +CPU= 32.940
  10:45:31 STEP 2746 H=2746:00 +CPU= 21.890
  10:45:53 STEP 2747 H=2747:00 +CPU= 21.021
  10:46:31 STEP 2748 H=2748:00 +CPU= 36.444
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002748
  10:46:54 STEP 2749 H=2749:00 +CPU= 21.245
  10:47:17 STEP 2750 H=2750:00 +CPU= 22.143
  10:47:53 STEP 2751 H=2751:00 +CPU= 33.148
  10:48:16 STEP 2752 H=2752:00 +CPU= 21.855
  10:48:40 STEP 2753 H=2753:00 +CPU= 21.674
  10:49:16 STEP 2754 H=2754:00 +CPU= 33.785
  10:49:37 STEP 2755 H=2755:00 +CPU= 20.553
  10:50:01 STEP 2756 H=2756:00 +CPU= 22.342
  10:50:36 STEP 2757 H=2757:00 +CPU= 32.631
  10:50:58 STEP 2758 H=2758:00 +CPU= 21.439
  10:51:21 STEP 2759 H=2759:00 +CPU= 22.045
  10:52:03 STEP 2760 H=2760:00 +CPU= 38.386
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_114.zip
Uploading the intermediate file: upload_file_114.zip
  10:52:25 STEP 2761 H=2761:00 +CPU= 21.081
  10:52:48 STEP 2762 H=2762:00 +CPU= 22.275
  10:53:24 STEP 2763 H=2763:00 +CPU= 33.192
  10:53:46 STEP 2764 H=2764:00 +CPU= 21.378
  10:54:09 STEP 2765 H=2765:00 +CPU= 21.765
  10:54:45 STEP 2766 H=2766:00 +CPU= 33.899
  10:55:08 STEP 2767 H=2767:00 +CPU= 20.982
  10:55:32 STEP 2768 H=2768:00 +CPU= 22.544
  10:56:07 STEP 2769 H=2769:00 +CPU= 32.993
  10:56:30 STEP 2770 H=2770:00 +CPU= 21.722
  10:56:54 STEP 2771 H=2771:00 +CPU= 21.833
  10:57:30 STEP 2772 H=2772:00 +CPU= 35.374
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002772
  10:57:52 STEP 2773 H=2773:00 +CPU= 20.509
  10:58:15 STEP 2774 H=2774:00 +CPU= 21.822
  10:58:49 STEP 2775 H=2775:00 +CPU= 32.409
  10:59:11 STEP 2776 H=2776:00 +CPU= 20.652
  10:59:32 STEP 2777 H=2777:00 +CPU= 20.260
  11:00:08 STEP 2778 H=2778:00 +CPU= 33.245
  11:00:30 STEP 2779 H=2779:00 +CPU= 20.856
  11:00:54 STEP 2780 H=2780:00 +CPU= 22.272
  11:01:29 STEP 2781 H=2781:00 +CPU= 32.570
  11:01:51 STEP 2782 H=2782:00 +CPU= 21.187
  11:02:13 STEP 2783 H=2783:00 +CPU= 20.182
  11:02:52 STEP 2784 H=2784:00 +CPU= 36.714
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_115.zip
Uploading the intermediate file: upload_file_115.zip
  11:03:13 STEP 2785 H=2785:00 +CPU= 20.219
  11:03:34 STEP 2786 H=2786:00 +CPU= 19.894
  11:04:06 STEP 2787 H=2787:00 +CPU= 30.878
  11:04:28 STEP 2788 H=2788:00 +CPU= 20.957
  11:04:50 STEP 2789 H=2789:00 +CPU= 20.563
  11:05:24 STEP 2790 H=2790:00 +CPU= 31.347
  11:05:45 STEP 2791 H=2791:00 +CPU= 19.359
  11:06:06 STEP 2792 H=2792:00 +CPU= 19.894
  11:06:39 STEP 2793 H=2793:00 +CPU= 30.591
  11:07:02 STEP 2794 H=2794:00 +CPU= 21.444
  11:07:23 STEP 2795 H=2795:00 +CPU= 20.219
  11:08:00 STEP 2796 H=2796:00 +CPU= 34.670
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002796
  11:08:19 STEP 2797 H=2797:00 +CPU= 19.009
  11:08:42 STEP 2798 H=2798:00 +CPU= 20.439
  11:09:15 STEP 2799 H=2799:00 +CPU= 31.032
  11:09:36 STEP 2800 H=2800:00 +CPU= 19.777
  11:09:57 STEP 2801 H=2801:00 +CPU= 18.979
  11:10:30 STEP 2802 H=2802:00 +CPU= 31.037
  11:10:52 STEP 2803 H=2803:00 +CPU= 20.407
  11:11:13 STEP 2804 H=2804:00 +CPU= 20.155
  11:11:44 STEP 2805 H=2805:00 +CPU= 29.402
  11:12:05 STEP 2806 H=2806:00 +CPU= 19.475
  11:12:26 STEP 2807 H=2807:00 +CPU= 20.456
  11:13:03 STEP 2808 H=2808:00 +CPU= 34.348
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_116.zip
Uploading the intermediate file: upload_file_116.zip
  11:13:23 STEP 2809 H=2809:00 +CPU= 18.751
  11:13:44 STEP 2810 H=2810:00 +CPU= 19.845
  11:14:18 STEP 2811 H=2811:00 +CPU= 31.375
  11:14:40 STEP 2812 H=2812:00 +CPU= 20.754
  11:15:02 STEP 2813 H=2813:00 +CPU= 19.424
  11:15:35 STEP 2814 H=2814:00 +CPU= 30.820
  11:15:56 STEP 2815 H=2815:00 +CPU= 19.793
  11:16:18 STEP 2816 H=2816:00 +CPU= 20.520
  11:16:50 STEP 2817 H=2817:00 +CPU= 30.512
  11:17:11 STEP 2818 H=2818:00 +CPU= 19.593
  11:17:31 STEP 2819 H=2819:00 +CPU= 18.746
  11:18:05 STEP 2820 H=2820:00 +CPU= 32.786
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002820
  11:18:27 STEP 2821 H=2821:00 +CPU= 20.124
  11:18:50 STEP 2822 H=2822:00 +CPU= 20.839
  11:19:23 STEP 2823 H=2823:00 +CPU= 31.101
  11:19:46 STEP 2824 H=2824:00 +CPU= 20.755
  11:20:08 STEP 2825 H=2825:00 +CPU= 20.119
  11:20:42 STEP 2826 H=2826:00 +CPU= 32.344
  11:21:04 STEP 2827 H=2827:00 +CPU= 19.974
  11:21:26 STEP 2828 H=2828:00 +CPU= 20.863
  11:21:59 STEP 2829 H=2829:00 +CPU= 31.438
  11:22:21 STEP 2830 H=2830:00 +CPU= 20.589
  11:22:42 STEP 2831 H=2831:00 +CPU= 20.686
  11:23:21 STEP 2832 H=2832:00 +CPU= 36.543
Moving to projects directory: /var/lib/boinc/slots/9/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/9/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/9/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12166822/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0178_1981050100_123_950_12166822_0_r1049599650_117.zip
Uploading the intermediate file: upload_file_117.zip
  11:23:42 STEP 2833 H=2833:00 +CPU= 20.163
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:47:41 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 9,676,800 79,579 0.0082
09 May 2023 15:47:30 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 8,812,800 73,066 0.0083
09 May 2023 15:47:18 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 7,948,800 66,421 0.0084
09 May 2023 15:47:06 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 7,084,800 59,416 0.0084
09 May 2023 15:46:54 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 6,220,800 52,442 0.0084
09 May 2023 15:46:44 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 5,356,800 45,543 0.0085
09 May 2023 15:46:30 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 4,492,800 38,498 0.0086
09 May 2023 15:46:20 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 3,628,800 31,209 0.0086
09 May 2023 15:46:09 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 2,764,800 23,732 0.0086
09 May 2023 15:45:58 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 1,900,800 16,283 0.0086
09 May 2023 15:45:48 1507812 22251033 oifs_43r3_ps_0178_1981050100_123_950_12166822_0 1,036,800 8,703 0.0084


©2024 climateprediction.net