climateprediction.net home page
Task 22277953

Task 22277953

Name oifs_43r3_ps_0582_2006050100_123_975_12192226_0
Workunit 12192226
Created 23 Dec 2022, 13:33:13 UTC
Sent 21 Jan 2023, 14:41:09 UTC
Report deadline 22 Mar 2023, 14:41:09 UTC
Received 27 Jan 2023, 17:36:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1505071
Run time 12 hours 43 min 35 sec
CPU time 11 hours 38 min 13 sec
Validate state Invalid
Credit 12,096.00
Device peak FLOPS 7.68 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,409.93 MB
Peak swap size 4,978.06 MB
Peak disk usage 3,145.95 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
dding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_93.zip
Uploading the intermediate file: upload_file_93.zip
  03:48:16 STEP 2257 H=2257:00 +CPU= 12.033
  03:48:27 STEP 2258 H=2258:00 +CPU= 11.149
  03:48:43 STEP 2259 H=2259:00 +CPU= 16.396
  03:48:54 STEP 2260 H=2260:00 +CPU= 11.029
  03:49:05 STEP 2261 H=2261:00 +CPU= 10.761
  03:49:23 STEP 2262 H=2262:00 +CPU= 17.211
  03:49:33 STEP 2263 H=2263:00 +CPU= 10.407
  03:49:44 STEP 2264 H=2264:00 +CPU= 10.731
  03:50:00 STEP 2265 H=2265:00 +CPU= 16.641
  03:50:11 STEP 2266 H=2266:00 +CPU= 10.732
  03:50:22 STEP 2267 H=2267:00 +CPU= 10.425
  03:50:40 STEP 2268 H=2268:00 +CPU= 18.642
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  03:50:51 STEP 2269 H=2269:00 +CPU= 10.410
  03:51:02 STEP 2270 H=2270:00 +CPU= 10.807
  03:51:18 STEP 2271 H=2271:00 +CPU= 16.641
  03:51:29 STEP 2272 H=2272:00 +CPU= 10.683
  03:51:39 STEP 2273 H=2273:00 +CPU= 10.336
  03:51:56 STEP 2274 H=2274:00 +CPU= 16.841
  03:52:07 STEP 2275 H=2275:00 +CPU= 10.369
  03:52:18 STEP 2276 H=2276:00 +CPU= 10.891
  03:52:34 STEP 2277 H=2277:00 +CPU= 16.654
  03:52:45 STEP 2278 H=2278:00 +CPU= 10.650
  03:52:56 STEP 2279 H=2279:00 +CPU= 10.665
  03:53:16 STEP 2280 H=2280:00 +CPU= 19.983
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_94.zip
Uploading the intermediate file: upload_file_94.zip
  03:53:26 STEP 2281 H=2281:00 +CPU= 10.540
  03:53:37 STEP 2282 H=2282:00 +CPU= 10.722
  03:53:54 STEP 2283 H=2283:00 +CPU= 16.746
  03:54:05 STEP 2284 H=2284:00 +CPU= 10.879
  03:54:15 STEP 2285 H=2285:00 +CPU= 10.441
  03:54:32 STEP 2286 H=2286:00 +CPU= 17.083
  03:54:43 STEP 2287 H=2287:00 +CPU= 10.429
  03:54:54 STEP 2288 H=2288:00 +CPU= 10.661
  03:55:10 STEP 2289 H=2289:00 +CPU= 16.555
  03:55:21 STEP 2290 H=2290:00 +CPU= 10.710
  03:55:31 STEP 2291 H=2291:00 +CPU= 10.386
  03:55:50 STEP 2292 H=2292:00 +CPU= 18.569
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002292
  03:56:00 STEP 2293 H=2293:00 +CPU= 10.434
  03:56:11 STEP 2294 H=2294:00 +CPU= 10.741
  03:56:28 STEP 2295 H=2295:00 +CPU= 16.618
  03:56:39 STEP 2296 H=2296:00 +CPU= 10.594
  03:56:49 STEP 2297 H=2297:00 +CPU= 10.351
  03:57:06 STEP 2298 H=2298:00 +CPU= 17.039
  03:57:16 STEP 2299 H=2299:00 +CPU= 10.286
  03:57:27 STEP 2300 H=2300:00 +CPU= 10.807
  03:57:44 STEP 2301 H=2301:00 +CPU= 16.638
  03:57:55 STEP 2302 H=2302:00 +CPU= 10.863
  03:58:06 STEP 2303 H=2303:00 +CPU= 10.783
  03:58:26 STEP 2304 H=2304:00 +CPU= 20.002
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_95.zip
Uploading the intermediate file: upload_file_95.zip
  03:58:36 STEP 2305 H=2305:00 +CPU= 10.319
  03:58:47 STEP 2306 H=2306:00 +CPU= 10.634
  03:59:03 STEP 2307 H=2307:00 +CPU= 16.784
  03:59:14 STEP 2308 H=2308:00 +CPU= 10.812
  03:59:25 STEP 2309 H=2309:00 +CPU= 10.461
  03:59:42 STEP 2310 H=2310:00 +CPU= 16.857
  03:59:52 STEP 2311 H=2311:00 +CPU= 10.339
  04:00:03 STEP 2312 H=2312:00 +CPU= 10.654
  04:00:19 STEP 2313 H=2313:00 +CPU= 16.587
  04:00:30 STEP 2314 H=2314:00 +CPU= 10.610
  04:00:41 STEP 2315 H=2315:00 +CPU= 10.541
  04:00:59 STEP 2316 H=2316:00 +CPU= 18.720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002316
  04:01:10 STEP 2317 H=2317:00 +CPU= 10.489
  04:01:21 STEP 2318 H=2318:00 +CPU= 10.653
  04:01:37 STEP 2319 H=2319:00 +CPU= 16.709
  04:01:48 STEP 2320 H=2320:00 +CPU= 10.786
  04:01:59 STEP 2321 H=2321:00 +CPU= 10.319
  04:02:15 STEP 2322 H=2322:00 +CPU= 16.802
  04:02:26 STEP 2323 H=2323:00 +CPU= 10.431
  04:02:37 STEP 2324 H=2324:00 +CPU= 10.866
  04:02:53 STEP 2325 H=2325:00 +CPU= 16.490
  04:03:04 STEP 2326 H=2326:00 +CPU= 10.690
  04:03:15 STEP 2327 H=2327:00 +CPU= 10.737
  04:03:35 STEP 2328 H=2328:00 +CPU= 20.262
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_96.zip
Uploading the intermediate file: upload_file_96.zip
  04:03:46 STEP 2329 H=2329:00 +CPU= 10.452
  04:03:57 STEP 2330 H=2330:00 +CPU= 10.941
  04:04:13 STEP 2331 H=2331:00 +CPU= 16.860
  04:04:24 STEP 2332 H=2332:00 +CPU= 10.688
  04:04:35 STEP 2333 H=2333:00 +CPU= 10.288
  04:04:52 STEP 2334 H=2334:00 +CPU= 17.081
  04:05:02 STEP 2335 H=2335:00 +CPU= 10.354
  04:05:13 STEP 2336 H=2336:00 +CPU= 10.612
  04:05:29 STEP 2337 H=2337:00 +CPU= 16.624
  04:05:40 STEP 2338 H=2338:00 +CPU= 10.717
  04:05:50 STEP 2339 H=2339:00 +CPU= 10.367
  04:06:09 STEP 2340 H=2340:00 +CPU= 18.738
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002340
  04:06:20 STEP 2341 H=2341:00 +CPU= 10.473
  04:06:33 STEP 2342 H=2342:00 +CPU= 12.988
  04:06:54 STEP 2343 H=2343:00 +CPU= 20.983
  04:07:07 STEP 2344 H=2344:00 +CPU= 13.275
  04:07:20 STEP 2345 H=2345:00 +CPU= 12.828
  04:07:41 STEP 2346 H=2346:00 +CPU= 20.721
  04:07:54 STEP 2347 H=2347:00 +CPU= 12.614
  04:08:07 STEP 2348 H=2348:00 +CPU= 12.968
  04:08:27 STEP 2349 H=2349:00 +CPU= 19.884
  04:08:40 STEP 2350 H=2350:00 +CPU= 12.797
  04:08:53 STEP 2351 H=2351:00 +CPU= 12.977
  04:09:17 STEP 2352 H=2352:00 +CPU= 23.898
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_97.zip
Uploading the intermediate file: upload_file_97.zip
  04:09:30 STEP 2353 H=2353:00 +CPU= 12.408
  04:09:43 STEP 2354 H=2354:00 +CPU= 12.979
  04:10:03 STEP 2355 H=2355:00 +CPU= 20.048
  04:10:16 STEP 2356 H=2356:00 +CPU= 13.111
  04:10:29 STEP 2357 H=2357:00 +CPU= 12.810
  04:10:50 STEP 2358 H=2358:00 +CPU= 20.881
  04:11:03 STEP 2359 H=2359:00 +CPU= 12.885
  04:11:17 STEP 2360 H=2360:00 +CPU= 13.265
  04:11:37 STEP 2361 H=2361:00 +CPU= 20.259
  04:11:50 STEP 2362 H=2362:00 +CPU= 13.128
  04:12:03 STEP 2363 H=2363:00 +CPU= 13.137
  04:12:26 STEP 2364 H=2364:00 +CPU= 22.776
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002364
  04:12:39 STEP 2365 H=2365:00 +CPU= 12.794
  04:12:53 STEP 2366 H=2366:00 +CPU= 13.242
  04:13:13 STEP 2367 H=2367:00 +CPU= 20.178
  04:13:26 STEP 2368 H=2368:00 +CPU= 13.324
  04:13:39 STEP 2369 H=2369:00 +CPU= 12.755
  04:14:00 STEP 2370 H=2370:00 +CPU= 20.722
  04:14:13 STEP 2371 H=2371:00 +CPU= 12.836
  04:14:27 STEP 2372 H=2372:00 +CPU= 13.409
  04:14:47 STEP 2373 H=2373:00 +CPU= 20.555
  04:15:01 STEP 2374 H=2374:00 +CPU= 13.217
  04:15:14 STEP 2375 H=2375:00 +CPU= 13.403
  04:15:39 STEP 2376 H=2376:00 +CPU= 24.643
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_98.zip
Uploading the intermediate file: upload_file_98.zip
  04:15:52 STEP 2377 H=2377:00 +CPU= 12.434
  04:16:05 STEP 2378 H=2378:00 +CPU= 13.090
  04:16:26 STEP 2379 H=2379:00 +CPU= 20.600
  04:16:39 STEP 2380 H=2380:00 +CPU= 13.298
  04:16:52 STEP 2381 H=2381:00 +CPU= 12.859
  04:17:13 STEP 2382 H=2382:00 +CPU= 20.782
  04:17:26 STEP 2383 H=2383:00 +CPU= 12.896
  04:17:39 STEP 2384 H=2384:00 +CPU= 13.160
  04:18:00 STEP 2385 H=2385:00 +CPU= 20.576
  04:18:13 STEP 2386 H=2386:00 +CPU= 13.195
  04:18:26 STEP 2387 H=2387:00 +CPU= 13.008
  04:18:49 STEP 2388 H=2388:00 +CPU= 22.861
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002388
  04:19:02 STEP 2389 H=2389:00 +CPU= 12.794
  04:19:16 STEP 2390 H=2390:00 +CPU= 13.541
  04:19:37 STEP 2391 H=2391:00 +CPU= 20.550
  04:19:50 STEP 2392 H=2392:00 +CPU= 13.336
  04:20:03 STEP 2393 H=2393:00 +CPU= 12.847
  04:20:24 STEP 2394 H=2394:00 +CPU= 20.881
  04:20:37 STEP 2395 H=2395:00 +CPU= 13.168
  04:20:51 STEP 2396 H=2396:00 +CPU= 13.047
  04:21:11 STEP 2397 H=2397:00 +CPU= 20.465
  04:21:24 STEP 2398 H=2398:00 +CPU= 13.091
  04:21:38 STEP 2399 H=2399:00 +CPU= 13.205
  04:22:02 STEP 2400 H=2400:00 +CPU= 24.487
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_99.zip
Uploading the intermediate file: upload_file_99.zip
  04:22:15 STEP 2401 H=2401:00 +CPU= 12.862
  04:22:29 STEP 2402 H=2402:00 +CPU= 13.085
Uploading trickle at timestep: 8640000
  04:22:49 STEP 2403 H=2403:00 +CPU= 20.521
  04:23:03 STEP 2404 H=2404:00 +CPU= 13.325
  04:23:16 STEP 2405 H=2405:00 +CPU= 13.152
  04:23:37 STEP 2406 H=2406:00 +CPU= 20.878
  04:23:50 STEP 2407 H=2407:00 +CPU= 13.081
  04:24:04 STEP 2408 H=2408:00 +CPU= 13.342
  04:24:25 STEP 2409 H=2409:00 +CPU= 20.722
  04:24:38 STEP 2410 H=2410:00 +CPU= 13.254
  04:24:51 STEP 2411 H=2411:00 +CPU= 13.140
  04:25:15 STEP 2412 H=2412:00 +CPU= 23.182
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002412
  04:25:28 STEP 2413 H=2413:00 +CPU= 12.938
  04:25:41 STEP 2414 H=2414:00 +CPU= 13.551
  04:26:02 STEP 2415 H=2415:00 +CPU= 20.637
  04:26:16 STEP 2416 H=2416:00 +CPU= 13.446
  04:26:29 STEP 2417 H=2417:00 +CPU= 12.842
  04:26:50 STEP 2418 H=2418:00 +CPU= 21.111
  04:27:03 STEP 2419 H=2419:00 +CPU= 12.876
  04:27:16 STEP 2420 H=2420:00 +CPU= 13.385
  04:27:37 STEP 2421 H=2421:00 +CPU= 20.347
  04:27:50 STEP 2422 H=2422:00 +CPU= 13.272
  04:28:04 STEP 2423 H=2423:00 +CPU= 13.097
  04:28:29 STEP 2424 H=2424:00 +CPU= 24.865
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_100.zip
Uploading the intermediate file: upload_file_100.zip
  04:28:41 STEP 2425 H=2425:00 +CPU= 12.583
  04:28:55 STEP 2426 H=2426:00 +CPU= 13.201
  04:29:16 STEP 2427 H=2427:00 +CPU= 20.819
  04:29:29 STEP 2428 H=2428:00 +CPU= 13.277
  04:29:42 STEP 2429 H=2429:00 +CPU= 12.966
  04:30:04 STEP 2430 H=2430:00 +CPU= 21.497
  04:30:17 STEP 2431 H=2431:00 +CPU= 12.749
  04:30:30 STEP 2432 H=2432:00 +CPU= 13.393
  04:30:51 STEP 2433 H=2433:00 +CPU= 20.707
  04:31:04 STEP 2434 H=2434:00 +CPU= 13.334
  04:31:18 STEP 2435 H=2435:00 +CPU= 12.930
  04:31:41 STEP 2436 H=2436:00 +CPU= 23.180
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002436
  04:31:54 STEP 2437 H=2437:00 +CPU= 13.038
  04:32:08 STEP 2438 H=2438:00 +CPU= 13.447
  04:32:28 STEP 2439 H=2439:00 +CPU= 20.555
  04:32:42 STEP 2440 H=2440:00 +CPU= 13.395
  04:32:55 STEP 2441 H=2441:00 +CPU= 12.728
  04:33:16 STEP 2442 H=2442:00 +CPU= 20.986
  04:33:29 STEP 2443 H=2443:00 +CPU= 12.830
  04:33:42 STEP 2444 H=2444:00 +CPU= 13.401
  04:34:03 STEP 2445 H=2445:00 +CPU= 20.662
  04:34:17 STEP 2446 H=2446:00 +CPU= 13.336
  04:34:30 STEP 2447 H=2447:00 +CPU= 13.357
  04:34:55 STEP 2448 H=2448:00 +CPU= 24.898
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_101.zip
Uploading the intermediate file: upload_file_101.zip
  04:35:08 STEP 2449 H=2449:00 +CPU= 13.021
  04:35:21 STEP 2450 H=2450:00 +CPU= 12.228
  04:35:39 STEP 2451 H=2451:00 +CPU= 18.186
  04:35:51 STEP 2452 H=2452:00 +CPU= 12.147
  04:36:03 STEP 2453 H=2453:00 +CPU= 11.604
  04:36:22 STEP 2454 H=2454:00 +CPU= 18.905
  04:36:34 STEP 2455 H=2455:00 +CPU= 12.003
  04:36:46 STEP 2456 H=2456:00 +CPU= 12.103
  04:37:05 STEP 2457 H=2457:00 +CPU= 18.726
  04:37:17 STEP 2458 H=2458:00 +CPU= 12.254
  04:37:29 STEP 2459 H=2459:00 +CPU= 11.736
  04:37:50 STEP 2460 H=2460:00 +CPU= 21.283
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  04:38:02 STEP 2461 H=2461:00 +CPU= 11.505
  04:38:14 STEP 2462 H=2462:00 +CPU= 11.801
  04:38:32 STEP 2463 H=2463:00 +CPU= 18.420
  04:38:45 STEP 2464 H=2464:00 +CPU= 12.525
  04:38:57 STEP 2465 H=2465:00 +CPU= 11.915
  04:39:17 STEP 2466 H=2466:00 +CPU= 19.779
  04:39:28 STEP 2467 H=2467:00 +CPU= 10.610
  04:39:39 STEP 2468 H=2468:00 +CPU= 11.125
  04:39:56 STEP 2469 H=2469:00 +CPU= 17.647
  04:40:09 STEP 2470 H=2470:00 +CPU= 12.400
  04:40:22 STEP 2471 H=2471:00 +CPU= 13.039
  04:40:44 STEP 2472 H=2472:00 +CPU= 21.609
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_102.zip
Uploading the intermediate file: upload_file_102.zip
  04:40:56 STEP 2473 H=2473:00 +CPU= 11.979
  04:41:09 STEP 2474 H=2474:00 +CPU= 13.346
  04:41:29 STEP 2475 H=2475:00 +CPU= 19.682
  04:41:42 STEP 2476 H=2476:00 +CPU= 12.726
  04:41:54 STEP 2477 H=2477:00 +CPU= 12.354
  04:42:15 STEP 2478 H=2478:00 +CPU= 20.171
  04:42:27 STEP 2479 H=2479:00 +CPU= 12.337
  04:42:40 STEP 2480 H=2480:00 +CPU= 12.992
  04:43:00 STEP 2481 H=2481:00 +CPU= 19.774
  04:43:13 STEP 2482 H=2482:00 +CPU= 12.910
  04:43:26 STEP 2483 H=2483:00 +CPU= 12.383
  04:43:48 STEP 2484 H=2484:00 +CPU= 21.974
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  04:44:00 STEP 2485 H=2485:00 +CPU= 12.389
  04:44:13 STEP 2486 H=2486:00 +CPU= 12.401
  04:44:32 STEP 2487 H=2487:00 +CPU= 18.892
  04:44:44 STEP 2488 H=2488:00 +CPU= 12.113
  04:44:56 STEP 2489 H=2489:00 +CPU= 12.325
  04:45:17 STEP 2490 H=2490:00 +CPU= 20.197
  04:45:29 STEP 2491 H=2491:00 +CPU= 12.839
  04:45:42 STEP 2492 H=2492:00 +CPU= 12.879
  04:46:03 STEP 2493 H=2493:00 +CPU= 20.165
  04:46:16 STEP 2494 H=2494:00 +CPU= 13.142
  04:46:29 STEP 2495 H=2495:00 +CPU= 13.112
  04:46:54 STEP 2496 H=2496:00 +CPU= 24.417
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_103.zip
Uploading the intermediate file: upload_file_103.zip
  04:47:07 STEP 2497 H=2497:00 +CPU= 12.858
  04:47:20 STEP 2498 H=2498:00 +CPU= 12.936
  04:47:40 STEP 2499 H=2499:00 +CPU= 20.325
  04:47:54 STEP 2500 H=2500:00 +CPU= 13.188
  04:48:07 STEP 2501 H=2501:00 +CPU= 12.736
  04:48:27 STEP 2502 H=2502:00 +CPU= 20.493
  04:48:40 STEP 2503 H=2503:00 +CPU= 12.828
  04:48:54 STEP 2504 H=2504:00 +CPU= 13.309
  04:49:15 STEP 2505 H=2505:00 +CPU= 20.973
  04:49:28 STEP 2506 H=2506:00 +CPU= 13.096
  04:49:41 STEP 2507 H=2507:00 +CPU= 12.667
  04:50:04 STEP 2508 H=2508:00 +CPU= 22.663
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  04:50:17 STEP 2509 H=2509:00 +CPU= 12.856
  04:50:30 STEP 2510 H=2510:00 +CPU= 12.948
  04:50:50 STEP 2511 H=2511:00 +CPU= 20.399
  04:51:03 STEP 2512 H=2512:00 +CPU= 13.150
  04:51:16 STEP 2513 H=2513:00 +CPU= 12.658
  04:51:37 STEP 2514 H=2514:00 +CPU= 20.807
  04:51:50 STEP 2515 H=2515:00 +CPU= 12.844
  04:52:03 STEP 2516 H=2516:00 +CPU= 13.147
  04:52:24 STEP 2517 H=2517:00 +CPU= 20.340
  04:52:37 STEP 2518 H=2518:00 +CPU= 13.072
  04:52:50 STEP 2519 H=2519:00 +CPU= 13.038
  04:53:15 STEP 2520 H=2520:00 +CPU= 24.386
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_104.zip
Uploading the intermediate file: upload_file_104.zip
  04:53:26 STEP 2521 H=2521:00 +CPU= 11.360
  04:53:37 STEP 2522 H=2522:00 +CPU= 10.955
  04:53:55 STEP 2523 H=2523:00 +CPU= 17.640
  04:54:08 STEP 2524 H=2524:00 +CPU= 13.484
  04:54:21 STEP 2525 H=2525:00 +CPU= 12.772
  04:54:42 STEP 2526 H=2526:00 +CPU= 20.746
  04:54:55 STEP 2527 H=2527:00 +CPU= 12.697
  04:55:09 STEP 2528 H=2528:00 +CPU= 13.581
  04:55:29 STEP 2529 H=2529:00 +CPU= 19.703
  04:55:42 STEP 2530 H=2530:00 +CPU= 13.473
  04:55:55 STEP 2531 H=2531:00 +CPU= 12.887
  04:56:18 STEP 2532 H=2532:00 +CPU= 22.864
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002532
  04:56:31 STEP 2533 H=2533:00 +CPU= 12.643
  04:56:44 STEP 2534 H=2534:00 +CPU= 12.994
  04:57:04 STEP 2535 H=2535:00 +CPU= 20.108
  04:57:17 STEP 2536 H=2536:00 +CPU= 12.982
  04:57:30 STEP 2537 H=2537:00 +CPU= 12.395
  04:57:51 STEP 2538 H=2538:00 +CPU= 20.523
  04:58:04 STEP 2539 H=2539:00 +CPU= 13.023
  04:58:17 STEP 2540 H=2540:00 +CPU= 12.837
  04:58:34 STEP 2541 H=2541:00 +CPU= 17.560
  04:58:48 STEP 2542 H=2542:00 +CPU= 13.588
  04:59:01 STEP 2543 H=2543:00 +CPU= 12.899
  04:59:25 STEP 2544 H=2544:00 +CPU= 24.271
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_105.zip
Uploading the intermediate file: upload_file_105.zip
  04:59:38 STEP 2545 H=2545:00 +CPU= 12.385
  04:59:51 STEP 2546 H=2546:00 +CPU= 12.906
  05:00:10 STEP 2547 H=2547:00 +CPU= 18.903
  05:00:22 STEP 2548 H=2548:00 +CPU= 11.885
  05:00:34 STEP 2549 H=2549:00 +CPU= 11.959
  05:00:53 STEP 2550 H=2550:00 +CPU= 18.740
  05:01:05 STEP 2551 H=2551:00 +CPU= 12.010
  05:01:17 STEP 2552 H=2552:00 +CPU= 11.924
  05:01:36 STEP 2553 H=2553:00 +CPU= 18.596
  05:01:48 STEP 2554 H=2554:00 +CPU= 12.039
  05:01:59 STEP 2555 H=2555:00 +CPU= 11.666
  05:02:21 STEP 2556 H=2556:00 +CPU= 21.123
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  05:02:33 STEP 2557 H=2557:00 +CPU= 11.998
  05:02:45 STEP 2558 H=2558:00 +CPU= 12.557
  05:03:04 STEP 2559 H=2559:00 +CPU= 18.434
  05:03:16 STEP 2560 H=2560:00 +CPU= 12.369
  05:03:28 STEP 2561 H=2561:00 +CPU= 12.017
  05:03:48 STEP 2562 H=2562:00 +CPU= 19.040
  05:04:00 STEP 2563 H=2563:00 +CPU= 12.424
  05:04:13 STEP 2564 H=2564:00 +CPU= 12.846
  05:04:32 STEP 2565 H=2565:00 +CPU= 18.481
  05:04:44 STEP 2566 H=2566:00 +CPU= 12.124
  05:04:56 STEP 2567 H=2567:00 +CPU= 12.205
  05:05:19 STEP 2568 H=2568:00 +CPU= 22.660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_106.zip
Uploading the intermediate file: upload_file_106.zip
  05:05:31 STEP 2569 H=2569:00 +CPU= 11.957
  05:05:43 STEP 2570 H=2570:00 +CPU= 11.804
  05:06:01 STEP 2571 H=2571:00 +CPU= 18.344
  05:06:14 STEP 2572 H=2572:00 +CPU= 12.297
  05:06:25 STEP 2573 H=2573:00 +CPU= 11.622
  05:06:44 STEP 2574 H=2574:00 +CPU= 18.664
  05:06:56 STEP 2575 H=2575:00 +CPU= 11.761
  05:07:08 STEP 2576 H=2576:00 +CPU= 12.187
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           31  MAX ETADOT VERTICAL VEL.=   1.662298076726781E-007
  LON  =    146.250000000000       degrees
  LAT  =   -35.3269057618723       degrees
  05:07:27 STEP 2577 H=2577:00 +CPU= 18.852
  05:07:40 STEP 2578 H=2578:00 +CPU= 12.479
  05:07:52 STEP 2579 H=2579:00 +CPU= 12.103
  05:08:14 STEP 2580 H=2580:00 +CPU= 21.917
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  05:08:27 STEP 2581 H=2581:00 +CPU= 12.973
  05:08:40 STEP 2582 H=2582:00 +CPU= 13.128
  05:09:01 STEP 2583 H=2583:00 +CPU= 20.349
  05:09:14 STEP 2584 H=2584:00 +CPU= 13.223
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=            1  MAX ETADOT VERTICAL VEL.=   2.055562648834013E-007
  LON  =    143.750000000000       degrees
  LAT  =   -38.6913661222016       degrees
  05:09:26 STEP 2585 H=2585:00 +CPU= 11.693
  05:09:45 STEP 2586 H=2586:00 +CPU= 19.191
  05:09:57 STEP 2587 H=2587:00 +CPU= 11.692
  05:10:09 STEP 2588 H=2588:00 +CPU= 12.241
  05:10:28 STEP 2589 H=2589:00 +CPU= 18.844
  05:10:40 STEP 2590 H=2590:00 +CPU= 11.973
  05:10:53 STEP 2591 H=2591:00 +CPU= 12.388
  05:11:15 STEP 2592 H=2592:00 +CPU= 22.691
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_107.zip
Uploading the intermediate file: upload_file_107.zip
  05:11:27 STEP 2593 H=2593:00 +CPU= 11.664
  05:11:39 STEP 2594 H=2594:00 +CPU= 12.200
  05:11:58 STEP 2595 H=2595:00 +CPU= 18.903
  05:12:11 STEP 2596 H=2596:00 +CPU= 12.452
  05:12:23 STEP 2597 H=2597:00 +CPU= 12.067
  05:12:42 STEP 2598 H=2598:00 +CPU= 19.142
  05:12:54 STEP 2599 H=2599:00 +CPU= 11.480
  05:13:06 STEP 2600 H=2600:00 +CPU= 11.730
  05:13:24 STEP 2601 H=2601:00 +CPU= 18.587
  05:13:37 STEP 2602 H=2602:00 +CPU= 12.358
  05:13:48 STEP 2603 H=2603:00 +CPU= 11.566
  05:14:10 STEP 2604 H=2604:00 +CPU= 21.342
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002604
  05:14:22 STEP 2605 H=2605:00 +CPU= 11.724
  05:14:34 STEP 2606 H=2606:00 +CPU= 12.254
  05:14:53 STEP 2607 H=2607:00 +CPU= 18.992
  05:15:06 STEP 2608 H=2608:00 +CPU= 12.409
  05:15:18 STEP 2609 H=2609:00 +CPU= 12.190
  05:15:38 STEP 2610 H=2610:00 +CPU= 19.599
  05:15:50 STEP 2611 H=2611:00 +CPU= 12.256
  05:16:02 STEP 2612 H=2612:00 +CPU= 11.940
  05:16:19 STEP 2613 H=2613:00 +CPU= 17.175
  05:16:30 STEP 2614 H=2614:00 +CPU= 10.932
  05:16:41 STEP 2615 H=2615:00 +CPU= 10.716
  05:17:01 STEP 2616 H=2616:00 +CPU= 20.017
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_108.zip
Uploading the intermediate file: upload_file_108.zip
  05:17:11 STEP 2617 H=2617:00 +CPU= 10.334
  05:17:22 STEP 2618 H=2618:00 +CPU= 10.760
  05:17:39 STEP 2619 H=2619:00 +CPU= 16.795
  05:17:50 STEP 2620 H=2620:00 +CPU= 10.794
  05:18:00 STEP 2621 H=2621:00 +CPU= 10.483
  05:18:17 STEP 2622 H=2622:00 +CPU= 17.206
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.068] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.085] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.091] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.092] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051824:1674364704:40691.098] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051825:1674364705:40692.100] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Turing:1:1:29372:29372] [20230122:051825:1674364705:40692.100] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FAABF7  vdfmain_                  923  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2006050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0582
(argv4) batchid: 975
(argv5) wuid: 12192226
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e995015ad986fa2d6aa5bf722b0a4ed8 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0582_2006050100_123_975_12192226.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0582_2006050100_123_975_12192226.zip
ic_ancil_file: ic_ancil_12192226
ifsdata_file: ifsdata_12192226
climate_data_file: clim_data_12192226
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2abb4b25874c83ab7bcf30e79c7898ac to: /var/lib/boinc-client/slots/1/ic_ancil_12192226.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12192226.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b97c6bdfa54fd3ad1b98db064aa47422 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12192226.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12192226.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12192226.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12192226.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12192226.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12192226.xml
-- Model is restarting --
Adjusting last_iter, 2623, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12192226.xml
last_cpu_time: 40298
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827
The child process has been launched with process id: 3657
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Turing:1:1:3657:3657] [20230127:170714:1674839234:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:07:35 STEP 2616 H=2616:00 +CPU= 20.224
  17:07:46 STEP 2617 H=2617:00 +CPU= 11.331
  17:07:58 STEP 2618 H=2618:00 +CPU= 11.718
  17:08:14 STEP 2619 H=2619:00 +CPU= 16.115
  17:08:24 STEP 2620 H=2620:00 +CPU= 10.172
  17:08:35 STEP 2621 H=2621:00 +CPU= 10.848
  17:08:52 STEP 2622 H=2622:00 +CPU= 16.256
  17:09:01 STEP 2623 H=2623:00 +CPU=  9.869
  17:09:12 STEP 2624 H=2624:00 +CPU= 10.160
  17:09:27 STEP 2625 H=2625:00 +CPU= 15.668
  17:09:37 STEP 2626 H=2626:00 +CPU= 10.074
  17:09:47 STEP 2627 H=2627:00 +CPU= 10.016
  17:10:06 STEP 2628 H=2628:00 +CPU= 18.343
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002628
  17:10:16 STEP 2629 H=2629:00 +CPU= 10.410
  17:10:27 STEP 2630 H=2630:00 +CPU= 10.488
  17:10:43 STEP 2631 H=2631:00 +CPU= 16.506
  17:10:54 STEP 2632 H=2632:00 +CPU= 10.598
  17:11:04 STEP 2633 H=2633:00 +CPU= 10.112
  17:11:20 STEP 2634 H=2634:00 +CPU= 16.072
  17:11:30 STEP 2635 H=2635:00 +CPU=  9.876
  17:11:40 STEP 2636 H=2636:00 +CPU= 10.068
  17:11:56 STEP 2637 H=2637:00 +CPU= 15.633
  17:12:06 STEP 2638 H=2638:00 +CPU= 10.104
  17:12:16 STEP 2639 H=2639:00 +CPU= 10.252
  17:12:37 STEP 2640 H=2640:00 +CPU= 20.292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_109.zip
Uploading the intermediate file: upload_file_109.zip
  17:12:47 STEP 2641 H=2641:00 +CPU= 10.691
  17:12:59 STEP 2642 H=2642:00 +CPU= 11.373
  17:13:16 STEP 2643 H=2643:00 +CPU= 17.568
  17:13:28 STEP 2644 H=2644:00 +CPU= 11.350
  17:13:38 STEP 2645 H=2645:00 +CPU= 10.494
  17:13:55 STEP 2646 H=2646:00 +CPU= 17.038
  17:14:06 STEP 2647 H=2647:00 +CPU= 10.534
  17:14:17 STEP 2648 H=2648:00 +CPU= 10.891
  17:14:34 STEP 2649 H=2649:00 +CPU= 16.578
  17:14:44 STEP 2650 H=2650:00 +CPU= 10.699
  17:14:55 STEP 2651 H=2651:00 +CPU= 10.548
  17:15:14 STEP 2652 H=2652:00 +CPU= 18.593
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002652
  17:15:24 STEP 2653 H=2653:00 +CPU= 10.482
  17:15:35 STEP 2654 H=2654:00 +CPU= 10.885
  17:15:52 STEP 2655 H=2655:00 +CPU= 16.584
  17:16:03 STEP 2656 H=2656:00 +CPU= 10.674
  17:16:13 STEP 2657 H=2657:00 +CPU= 10.520
  17:16:31 STEP 2658 H=2658:00 +CPU= 17.303
  17:16:41 STEP 2659 H=2659:00 +CPU= 10.611
  17:16:52 STEP 2660 H=2660:00 +CPU= 10.738
  17:17:09 STEP 2661 H=2661:00 +CPU= 17.263
  17:17:21 STEP 2662 H=2662:00 +CPU= 11.517
  17:17:32 STEP 2663 H=2663:00 +CPU= 11.285
  17:17:55 STEP 2664 H=2664:00 +CPU= 22.260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_110.zip
Uploading the intermediate file: upload_file_110.zip
  17:18:07 STEP 2665 H=2665:00 +CPU= 11.967
  17:18:20 STEP 2666 H=2666:00 +CPU= 12.596
  17:18:38 STEP 2667 H=2667:00 +CPU= 18.495
  17:18:51 STEP 2668 H=2668:00 +CPU= 12.679
  17:19:03 STEP 2669 H=2669:00 +CPU= 11.787
  17:19:22 STEP 2670 H=2670:00 +CPU= 19.388
  17:19:35 STEP 2671 H=2671:00 +CPU= 12.788
  17:19:47 STEP 2672 H=2672:00 +CPU= 11.796
  17:20:06 STEP 2673 H=2673:00 +CPU= 18.856
  17:20:18 STEP 2674 H=2674:00 +CPU= 11.674
  17:20:29 STEP 2675 H=2675:00 +CPU= 11.061
  17:20:52 STEP 2676 H=2676:00 +CPU= 22.658
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002676
  17:21:05 STEP 2677 H=2677:00 +CPU= 12.819
  17:21:17 STEP 2678 H=2678:00 +CPU= 12.531
  17:21:38 STEP 2679 H=2679:00 +CPU= 20.128
  17:21:51 STEP 2680 H=2680:00 +CPU= 12.698
  17:22:03 STEP 2681 H=2681:00 +CPU= 12.126
  17:22:22 STEP 2682 H=2682:00 +CPU= 19.109
  17:22:35 STEP 2683 H=2683:00 +CPU= 12.298
  17:22:47 STEP 2684 H=2684:00 +CPU= 12.726
  17:23:07 STEP 2685 H=2685:00 +CPU= 19.743
  17:23:20 STEP 2686 H=2686:00 +CPU= 13.066
  17:23:34 STEP 2687 H=2687:00 +CPU= 12.936
  17:23:58 STEP 2688 H=2688:00 +CPU= 24.173
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_111.zip
Uploading the intermediate file: upload_file_111.zip
  17:24:10 STEP 2689 H=2689:00 +CPU= 12.179
  17:24:23 STEP 2690 H=2690:00 +CPU= 12.987
  17:24:44 STEP 2691 H=2691:00 +CPU= 20.136
  17:24:57 STEP 2692 H=2692:00 +CPU= 13.278
  17:25:11 STEP 2693 H=2693:00 +CPU= 13.448
  17:25:32 STEP 2694 H=2694:00 +CPU= 21.020
  17:25:45 STEP 2695 H=2695:00 +CPU= 13.095
  17:25:58 STEP 2696 H=2696:00 +CPU= 13.227
  17:26:20 STEP 2697 H=2697:00 +CPU= 20.990
  17:26:33 STEP 2698 H=2698:00 +CPU= 13.089
  17:26:46 STEP 2699 H=2699:00 +CPU= 12.572
  17:27:09 STEP 2700 H=2700:00 +CPU= 23.053
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002700
  17:27:21 STEP 2701 H=2701:00 +CPU= 11.764
  17:27:32 STEP 2702 H=2702:00 +CPU= 11.582
  17:27:52 STEP 2703 H=2703:00 +CPU= 19.451
  17:28:06 STEP 2704 H=2704:00 +CPU= 13.572
  17:28:19 STEP 2705 H=2705:00 +CPU= 13.144
  17:28:37 STEP 2706 H=2706:00 +CPU= 17.859
  17:28:47 STEP 2707 H=2707:00 +CPU= 10.457
  17:28:58 STEP 2708 H=2708:00 +CPU= 10.977
  17:29:15 STEP 2709 H=2709:00 +CPU= 16.678
  17:29:26 STEP 2710 H=2710:00 +CPU= 10.573
  17:29:37 STEP 2711 H=2711:00 +CPU= 10.800
  17:29:57 STEP 2712 H=2712:00 +CPU= 20.123
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12192226/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0582_2006050100_123_975_12192226_0_r1387699827_112.zip
Uploading the intermediate file: upload_file_112.zip
  17:30:07 STEP 2713 H=2713:00 +CPU= 10.576
  17:30:18 STEP 2714 H=2714:00 +CPU= 10.635
  17:30:35 STEP 2715 H=2715:00 +CPU= 16.843
  17:30:46 STEP 2716 H=2716:00 +CPU= 10.952
  17:30:57 STEP 2717 H=2717:00 +CPU= 10.578
  17:31:14 STEP 2718 H=2718:00 +CPU= 17.197
  17:31:25 STEP 2719 H=2719:00 +CPU= 10.756
  17:31:36 STEP 2720 H=2720:00 +CPU= 10.782
  17:31:53 STEP 2721 H=2721:00 +CPU= 17.068
  17:32:04 STEP 2722 H=2722:00 +CPU= 11.160
  17:32:15 STEP 2723 H=2723:00 +CPU= 10.848
  17:32:34 STEP 2724 H=2724:00 +CPU= 18.904
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002724
  17:32:44 STEP 2725 H=2725:00 +CPU= 10.496
  17:32:55 STEP 2726 H=2726:00 +CPU= 11.052
  17:33:12 STEP 2727 H=2727:00 +CPU= 16.784
  17:33:23 STEP 2728 H=2728:00 +CPU= 10.831
  17:33:33 STEP 2729 H=2729:00 +CPU= 10.196
  17:33:50 STEP 2730 H=2730:00 +CPU= 16.729
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:34:46 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 8,640,000 36,950 0.0043
09 May 2023 16:34:39 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 7,776,000 33,538 0.0043
09 May 2023 16:34:34 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 6,912,000 29,938 0.0043
09 May 2023 16:34:28 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 6,048,000 26,170 0.0043
09 May 2023 16:34:21 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 5,184,000 22,270 0.0043
09 May 2023 16:34:14 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 4,320,000 18,236 0.0042
09 May 2023 16:34:08 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 3,456,000 14,484 0.0042
09 May 2023 16:34:02 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 2,592,000 10,956 0.0042
09 May 2023 16:33:56 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 1,728,000 7,532 0.0044
09 May 2023 16:33:51 1505071 22277953 oifs_43r3_ps_0582_2006050100_123_975_12192226_0 864,000 3,769 0.0044


©2024 climateprediction.net