climateprediction.net home page
Task 22446086

Task 22446086

Name oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0
Workunit 12292480
Created 13 Jun 2024, 2:02:31 UTC
Sent 13 Jun 2024, 11:15:51 UTC
Report deadline 12 Aug 2024, 11:15:51 UTC
Received 13 Jun 2024, 22:35:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1539347
Run time 11 hours 17 min 11 sec
CPU time 11 hours 13 min 17 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 5.61 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.48 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,356.74 MB
Stderr
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
480/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0_r193802484_12.zip
Uploading the intermediate file: upload_file_12.zip
  12:45:15 STEP 1249 H= 312:15 +CPU= 24.487
  12:45:39 STEP 1250 H= 312:30 +CPU= 24.201
  12:46:04 STEP 1251 H= 312:45 +CPU= 24.528
  12:46:29 STEP 1252 H= 313:00 +CPU= 24.737
  12:46:53 STEP 1253 H= 313:15 +CPU= 24.474
  12:47:18 STEP 1254 H= 313:30 +CPU= 24.481
  12:47:42 STEP 1255 H= 313:45 +CPU= 24.477
  12:48:07 STEP 1256 H= 314:00 +CPU= 24.723
  12:48:32 STEP 1257 H= 314:15 +CPU= 24.552
  12:48:56 STEP 1258 H= 314:30 +CPU= 24.477
  12:49:21 STEP 1259 H= 314:45 +CPU= 24.822
  12:49:51 STEP 1260 H= 315:00 +CPU= 30.002
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001260
  12:50:16 STEP 1261 H= 315:15 +CPU= 24.749
  12:50:41 STEP 1262 H= 315:30 +CPU= 24.645
  12:51:06 STEP 1263 H= 315:45 +CPU= 24.674
  12:51:31 STEP 1264 H= 316:00 +CPU= 24.901
  12:51:55 STEP 1265 H= 316:15 +CPU= 24.478
  12:52:20 STEP 1266 H= 316:30 +CPU= 24.739
  12:52:45 STEP 1267 H= 316:45 +CPU= 24.643
  12:53:10 STEP 1268 H= 317:00 +CPU= 24.919
  12:53:35 STEP 1269 H= 317:15 +CPU= 24.570
  12:53:59 STEP 1270 H= 317:30 +CPU= 24.251
  12:54:24 STEP 1271 H= 317:45 +CPU= 24.672
  12:54:54 STEP 1272 H= 318:00 +CPU= 30.111
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001272
  12:55:19 STEP 1273 H= 318:15 +CPU= 24.570
  12:55:43 STEP 1274 H= 318:30 +CPU= 24.523
  12:56:08 STEP 1275 H= 318:45 +CPU= 24.703
  12:56:33 STEP 1276 H= 319:00 +CPU= 24.984
  12:56:58 STEP 1277 H= 319:15 +CPU= 24.659
  12:57:22 STEP 1278 H= 319:30 +CPU= 24.535
  12:57:47 STEP 1279 H= 319:45 +CPU= 24.569
  12:58:12 STEP 1280 H= 320:00 +CPU= 25.126
  12:58:37 STEP 1281 H= 320:15 +CPU= 24.425
  12:59:01 STEP 1282 H= 320:30 +CPU= 24.432
  12:59:26 STEP 1283 H= 320:45 +CPU= 24.809
  12:59:57 STEP 1284 H= 321:00 +CPU= 30.293
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001284
  13:00:22 STEP 1285 H= 321:15 +CPU= 24.790
  13:00:47 STEP 1286 H= 321:30 +CPU= 24.758
  13:01:11 STEP 1287 H= 321:45 +CPU= 24.552
  13:01:36 STEP 1288 H= 322:00 +CPU= 24.837
  13:02:01 STEP 1289 H= 322:15 +CPU= 24.543
  13:02:25 STEP 1290 H= 322:30 +CPU= 24.623
  13:02:50 STEP 1291 H= 322:45 +CPU= 24.565
  13:03:15 STEP 1292 H= 323:00 +CPU= 24.725
  13:03:40 STEP 1293 H= 323:15 +CPU= 24.503
  13:04:04 STEP 1294 H= 323:30 +CPU= 24.660
  13:04:29 STEP 1295 H= 323:45 +CPU= 24.466
  13:05:03 STEP 1296 H= 324:00 +CPU= 33.982
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001296
  13:05:28 STEP 1297 H= 324:15 +CPU= 24.586
  13:05:52 STEP 1298 H= 324:30 +CPU= 24.616
  13:06:17 STEP 1299 H= 324:45 +CPU= 24.702
  13:06:42 STEP 1300 H= 325:00 +CPU= 24.956
  13:07:07 STEP 1301 H= 325:15 +CPU= 24.662
  13:07:32 STEP 1302 H= 325:30 +CPU= 24.673
  13:07:56 STEP 1303 H= 325:45 +CPU= 24.671
  13:08:21 STEP 1304 H= 326:00 +CPU= 24.836
  13:08:46 STEP 1305 H= 326:15 +CPU= 24.740
  13:09:11 STEP 1306 H= 326:30 +CPU= 24.723
  13:09:36 STEP 1307 H= 326:45 +CPU= 24.549
  13:10:06 STEP 1308 H= 327:00 +CPU= 29.876
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001308
  13:10:30 STEP 1309 H= 327:15 +CPU= 24.487
  13:10:55 STEP 1310 H= 327:30 +CPU= 24.580
  13:11:20 STEP 1311 H= 327:45 +CPU= 24.624
  13:11:45 STEP 1312 H= 328:00 +CPU= 24.899
  13:12:09 STEP 1313 H= 328:15 +CPU= 24.481
  13:12:34 STEP 1314 H= 328:30 +CPU= 24.733
  13:12:59 STEP 1315 H= 328:45 +CPU= 24.717
  13:13:24 STEP 1316 H= 329:00 +CPU= 24.878
  13:13:48 STEP 1317 H= 329:15 +CPU= 24.547
  13:14:13 STEP 1318 H= 329:30 +CPU= 24.382
  13:14:37 STEP 1319 H= 329:45 +CPU= 24.450
  13:15:07 STEP 1320 H= 330:00 +CPU= 30.064
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001320
  13:15:32 STEP 1321 H= 330:15 +CPU= 24.721
  13:15:57 STEP 1322 H= 330:30 +CPU= 24.567
  13:16:22 STEP 1323 H= 330:45 +CPU= 24.741
  13:16:47 STEP 1324 H= 331:00 +CPU= 25.059
  13:17:12 STEP 1325 H= 331:15 +CPU= 24.573
  13:17:36 STEP 1326 H= 331:30 +CPU= 24.479
  13:18:01 STEP 1327 H= 331:45 +CPU= 24.514
  13:18:26 STEP 1328 H= 332:00 +CPU= 24.832
  13:18:50 STEP 1329 H= 332:15 +CPU= 24.597
  13:19:15 STEP 1330 H= 332:30 +CPU= 24.370
  13:19:40 STEP 1331 H= 332:45 +CPU= 24.678
  13:20:10 STEP 1332 H= 333:00 +CPU= 30.259
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001332
  13:20:35 STEP 1333 H= 333:15 +CPU= 24.647
  13:20:59 STEP 1334 H= 333:30 +CPU= 24.716
  13:21:24 STEP 1335 H= 333:45 +CPU= 24.693
  13:21:49 STEP 1336 H= 334:00 +CPU= 24.879
  13:22:14 STEP 1337 H= 334:15 +CPU= 24.629
  13:22:39 STEP 1338 H= 334:30 +CPU= 24.488
  13:23:03 STEP 1339 H= 334:45 +CPU= 24.576
  13:23:28 STEP 1340 H= 335:00 +CPU= 24.895
  13:23:53 STEP 1341 H= 335:15 +CPU= 24.446
  13:24:17 STEP 1342 H= 335:30 +CPU= 24.581
  13:24:42 STEP 1343 H= 335:45 +CPU= 24.556
  13:25:16 STEP 1344 H= 336:00 +CPU= 33.812
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0_r193802484_13.zip
Uploading the intermediate file: upload_file_13.zip
  13:25:41 STEP 1345 H= 336:15 +CPU= 24.608
  13:26:05 STEP 1346 H= 336:30 +CPU= 24.616
  13:26:30 STEP 1347 H= 336:45 +CPU= 24.865
  13:26:56 STEP 1348 H= 337:00 +CPU= 25.107
  13:27:20 STEP 1349 H= 337:15 +CPU= 24.767
  13:27:45 STEP 1350 H= 337:30 +CPU= 24.287
  13:28:09 STEP 1351 H= 337:45 +CPU= 24.352
  13:28:34 STEP 1352 H= 338:00 +CPU= 24.951
  13:28:59 STEP 1353 H= 338:15 +CPU= 24.551
  13:29:24 STEP 1354 H= 338:30 +CPU= 24.617
  13:29:48 STEP 1355 H= 338:45 +CPU= 24.778
  13:30:19 STEP 1356 H= 339:00 +CPU= 30.032
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001356
  13:30:43 STEP 1357 H= 339:15 +CPU= 24.418
  13:31:08 STEP 1358 H= 339:30 +CPU= 24.456
  13:31:32 STEP 1359 H= 339:45 +CPU= 24.420
  13:31:57 STEP 1360 H= 340:00 +CPU= 24.860
  13:32:22 STEP 1361 H= 340:15 +CPU= 24.495
  13:32:46 STEP 1362 H= 340:30 +CPU= 24.542
  13:33:11 STEP 1363 H= 340:45 +CPU= 24.576
  13:33:36 STEP 1364 H= 341:00 +CPU= 24.640
  13:34:00 STEP 1365 H= 341:15 +CPU= 24.637
  13:34:25 STEP 1366 H= 341:30 +CPU= 24.438
  13:34:49 STEP 1367 H= 341:45 +CPU= 24.356
  13:35:20 STEP 1368 H= 342:00 +CPU= 30.211
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001368
  13:35:45 STEP 1369 H= 342:15 +CPU= 24.746
  13:36:09 STEP 1370 H= 342:30 +CPU= 24.624
  13:36:34 STEP 1371 H= 342:45 +CPU= 24.634
  13:36:59 STEP 1372 H= 343:00 +CPU= 25.023
  13:37:24 STEP 1373 H= 343:15 +CPU= 24.583
  13:37:49 STEP 1374 H= 343:30 +CPU= 24.723
  13:38:13 STEP 1375 H= 343:45 +CPU= 24.617
  13:38:38 STEP 1376 H= 344:00 +CPU= 24.887
  13:39:03 STEP 1377 H= 344:15 +CPU= 24.471
  13:39:28 STEP 1378 H= 344:30 +CPU= 24.654
  13:39:52 STEP 1379 H= 344:45 +CPU= 24.605
  13:40:22 STEP 1380 H= 345:00 +CPU= 30.108
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001380
  13:40:47 STEP 1381 H= 345:15 +CPU= 24.739
  13:41:12 STEP 1382 H= 345:30 +CPU= 24.560
  13:41:37 STEP 1383 H= 345:45 +CPU= 24.455
  13:42:01 STEP 1384 H= 346:00 +CPU= 24.779
  13:42:26 STEP 1385 H= 346:15 +CPU= 24.568
  13:42:51 STEP 1386 H= 346:30 +CPU= 24.522
  13:43:15 STEP 1387 H= 346:45 +CPU= 24.560
  13:43:40 STEP 1388 H= 347:00 +CPU= 25.046
  13:44:05 STEP 1389 H= 347:15 +CPU= 24.443
  13:44:30 STEP 1390 H= 347:30 +CPU= 24.688
  13:44:54 STEP 1391 H= 347:45 +CPU= 24.634
  13:45:29 STEP 1392 H= 348:00 +CPU= 34.074
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001392
  13:45:54 STEP 1393 H= 348:15 +CPU= 24.873
  13:46:18 STEP 1394 H= 348:30 +CPU= 24.625
  13:46:43 STEP 1395 H= 348:45 +CPU= 24.446
  13:47:08 STEP 1396 H= 349:00 +CPU= 24.742
  13:47:32 STEP 1397 H= 349:15 +CPU= 24.492
  13:47:57 STEP 1398 H= 349:30 +CPU= 24.424
  13:48:21 STEP 1399 H= 349:45 +CPU= 24.534
  13:48:47 STEP 1400 H= 350:00 +CPU= 24.955
  13:49:11 STEP 1401 H= 350:15 +CPU= 24.597
  13:49:36 STEP 1402 H= 350:30 +CPU= 24.846
  13:50:01 STEP 1403 H= 350:45 +CPU= 24.354
  13:50:31 STEP 1404 H= 351:00 +CPU= 29.960
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001404
  13:50:55 STEP 1405 H= 351:15 +CPU= 24.674
  13:51:20 STEP 1406 H= 351:30 +CPU= 24.531
  13:51:45 STEP 1407 H= 351:45 +CPU= 24.502
  13:52:10 STEP 1408 H= 352:00 +CPU= 25.009
  13:52:35 STEP 1409 H= 352:15 +CPU= 24.818
  13:52:59 STEP 1410 H= 352:30 +CPU= 24.745
  13:53:24 STEP 1411 H= 352:45 +CPU= 24.479
  13:53:49 STEP 1412 H= 353:00 +CPU= 25.123
  13:54:14 STEP 1413 H= 353:15 +CPU= 24.465
  13:54:38 STEP 1414 H= 353:30 +CPU= 24.526
  13:55:03 STEP 1415 H= 353:45 +CPU= 24.467
  13:55:33 STEP 1416 H= 354:00 +CPU= 30.231
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001416
  13:55:58 STEP 1417 H= 354:15 +CPU= 24.506
  13:56:23 STEP 1418 H= 354:30 +CPU= 24.583
  13:56:47 STEP 1419 H= 354:45 +CPU= 24.276
  13:57:12 STEP 1420 H= 355:00 +CPU= 24.739
  13:57:36 STEP 1421 H= 355:15 +CPU= 24.459
  13:58:01 STEP 1422 H= 355:30 +CPU= 24.572
  13:58:26 STEP 1423 H= 355:45 +CPU= 24.637
  13:58:51 STEP 1424 H= 356:00 +CPU= 24.757
  13:59:15 STEP 1425 H= 356:15 +CPU= 24.321
  13:59:39 STEP 1426 H= 356:30 +CPU= 24.350
  14:00:04 STEP 1427 H= 356:45 +CPU= 24.359
  14:00:34 STEP 1428 H= 357:00 +CPU= 29.988
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001428
  14:00:59 STEP 1429 H= 357:15 +CPU= 24.815
  14:01:23 STEP 1430 H= 357:30 +CPU= 24.511
  14:01:48 STEP 1431 H= 357:45 +CPU= 24.669
  14:02:13 STEP 1432 H= 358:00 +CPU= 25.170
  14:02:38 STEP 1433 H= 358:15 +CPU= 24.686
  14:03:03 STEP 1434 H= 358:30 +CPU= 24.583
  14:03:28 STEP 1435 H= 358:45 +CPU= 24.766
  14:03:53 STEP 1436 H= 359:00 +CPU= 24.912
  14:04:17 STEP 1437 H= 359:15 +CPU= 24.549
  14:04:42 STEP 1438 H= 359:30 +CPU= 24.706
  14:05:07 STEP 1439 H= 359:45 +CPU= 24.651
  14:05:41 STEP 1440 H= 360:00 +CPU= 33.737
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0_r193802484_14.zip
Uploading the intermediate file: upload_file_14.zip
  14:06:05 STEP 1441 H= 360:15 +CPU= 24.566
  14:06:30 STEP 1442 H= 360:30 +CPU= 24.414
  14:06:55 STEP 1443 H= 360:45 +CPU= 24.795
  14:07:20 STEP 1444 H= 361:00 +CPU= 24.879
  14:07:44 STEP 1445 H= 361:15 +CPU= 24.513
  14:08:09 STEP 1446 H= 361:30 +CPU= 24.451
  14:08:33 STEP 1447 H= 361:45 +CPU= 24.516
  14:08:58 STEP 1448 H= 362:00 +CPU= 24.902
  14:09:23 STEP 1449 H= 362:15 +CPU= 24.753
  14:09:48 STEP 1450 H= 362:30 +CPU= 24.666
  14:10:13 STEP 1451 H= 362:45 +CPU= 24.720
  14:10:43 STEP 1452 H= 363:00 +CPU= 30.033
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001452
  14:11:08 STEP 1453 H= 363:15 +CPU= 24.571
  14:11:32 STEP 1454 H= 363:30 +CPU= 24.706
  14:11:57 STEP 1455 H= 363:45 +CPU= 24.420
  14:12:22 STEP 1456 H= 364:00 +CPU= 24.932
  14:12:47 STEP 1457 H= 364:15 +CPU= 24.647
  14:13:11 STEP 1458 H= 364:30 +CPU= 24.635
  14:13:36 STEP 1459 H= 364:45 +CPU= 24.578
  14:14:01 STEP 1460 H= 365:00 +CPU= 25.106
  14:14:26 STEP 1461 H= 365:15 +CPU= 24.585
  14:14:51 STEP 1462 H= 365:30 +CPU= 24.627
  14:15:15 STEP 1463 H= 365:45 +CPU= 24.529
  14:15:45 STEP 1464 H= 366:00 +CPU= 30.057
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001464
  14:16:10 STEP 1465 H= 366:15 +CPU= 24.598
  14:16:35 STEP 1466 H= 366:30 +CPU= 24.528
  14:17:00 STEP 1467 H= 366:45 +CPU= 24.644
  14:17:24 STEP 1468 H= 367:00 +CPU= 24.794
  14:17:49 STEP 1469 H= 367:15 +CPU= 24.400
  14:18:13 STEP 1470 H= 367:30 +CPU= 24.465
  14:18:38 STEP 1471 H= 367:45 +CPU= 24.337
  14:19:03 STEP 1472 H= 368:00 +CPU= 24.965
  14:19:28 STEP 1473 H= 368:15 +CPU= 24.561
  14:19:52 STEP 1474 H= 368:30 +CPU= 24.499
  14:20:16 STEP 1475 H= 368:45 +CPU= 24.190
  14:20:47 STEP 1476 H= 369:00 +CPU= 29.972
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001476
  14:21:11 STEP 1477 H= 369:15 +CPU= 24.545
  14:21:36 STEP 1478 H= 369:30 +CPU= 24.575
  14:22:00 STEP 1479 H= 369:45 +CPU= 24.487
  14:22:25 STEP 1480 H= 370:00 +CPU= 24.850
  14:22:50 STEP 1481 H= 370:15 +CPU= 24.490
  14:23:15 STEP 1482 H= 370:30 +CPU= 24.665
  14:23:39 STEP 1483 H= 370:45 +CPU= 24.475
  14:24:04 STEP 1484 H= 371:00 +CPU= 24.837
  14:24:29 STEP 1485 H= 371:15 +CPU= 24.435
  14:24:53 STEP 1486 H= 371:30 +CPU= 24.382
  14:25:18 STEP 1487 H= 371:45 +CPU= 24.476
  14:25:52 STEP 1488 H= 372:00 +CPU= 33.835
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001488
  14:26:16 STEP 1489 H= 372:15 +CPU= 24.535
  14:26:41 STEP 1490 H= 372:30 +CPU= 24.428
  14:27:05 STEP 1491 H= 372:45 +CPU= 24.321
  14:27:30 STEP 1492 H= 373:00 +CPU= 24.686
  14:27:55 STEP 1493 H= 373:15 +CPU= 24.541
  14:28:20 STEP 1494 H= 373:30 +CPU= 24.692
  14:28:44 STEP 1495 H= 373:45 +CPU= 24.645
  14:29:09 STEP 1496 H= 374:00 +CPU= 25.019
  14:29:34 STEP 1497 H= 374:15 +CPU= 24.459
  14:29:59 STEP 1498 H= 374:30 +CPU= 24.600
  14:30:23 STEP 1499 H= 374:45 +CPU= 24.520
  14:30:53 STEP 1500 H= 375:00 +CPU= 29.893
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001500
  14:31:18 STEP 1501 H= 375:15 +CPU= 24.710
  14:31:43 STEP 1502 H= 375:30 +CPU= 24.547
  14:32:07 STEP 1503 H= 375:45 +CPU= 24.591
  14:32:32 STEP 1504 H= 376:00 +CPU= 24.722
  14:32:57 STEP 1505 H= 376:15 +CPU= 24.393
  14:33:21 STEP 1506 H= 376:30 +CPU= 24.364
  14:33:46 STEP 1507 H= 376:45 +CPU= 24.454
  14:34:10 STEP 1508 H= 377:00 +CPU= 24.539
  14:34:35 STEP 1509 H= 377:15 +CPU= 24.262
  14:34:59 STEP 1510 H= 377:30 +CPU= 24.496
  14:35:24 STEP 1511 H= 377:45 +CPU= 24.509
  14:35:54 STEP 1512 H= 378:00 +CPU= 30.201
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001512
  14:36:19 STEP 1513 H= 378:15 +CPU= 24.499
  14:36:43 STEP 1514 H= 378:30 +CPU= 24.505
  14:37:08 STEP 1515 H= 378:45 +CPU= 24.457
  14:37:33 STEP 1516 H= 379:00 +CPU= 24.848
  14:37:57 STEP 1517 H= 379:15 +CPU= 24.417
  14:38:22 STEP 1518 H= 379:30 +CPU= 24.279
  14:38:46 STEP 1519 H= 379:45 +CPU= 24.433
  14:39:11 STEP 1520 H= 380:00 +CPU= 24.897
  14:39:36 STEP 1521 H= 380:15 +CPU= 24.358
  14:40:00 STEP 1522 H= 380:30 +CPU= 24.496
  14:40:25 STEP 1523 H= 380:45 +CPU= 24.423
  14:40:55 STEP 1524 H= 381:00 +CPU= 29.843
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001524
  14:41:19 STEP 1525 H= 381:15 +CPU= 24.413
  14:41:44 STEP 1526 H= 381:30 +CPU= 24.456
  14:42:09 STEP 1527 H= 381:45 +CPU= 24.805
  14:42:33 STEP 1528 H= 382:00 +CPU= 24.710
  14:42:58 STEP 1529 H= 382:15 +CPU= 24.541
  14:43:23 STEP 1530 H= 382:30 +CPU= 24.591
  14:43:47 STEP 1531 H= 382:45 +CPU= 24.417
  14:44:12 STEP 1532 H= 383:00 +CPU= 24.873
  14:44:37 STEP 1533 H= 383:15 +CPU= 24.455
  14:45:01 STEP 1534 H= 383:30 +CPU= 24.466
  14:45:26 STEP 1535 H= 383:45 +CPU= 24.430
  14:46:00 STEP 1536 H= 384:00 +CPU= 33.848
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0_r193802484_15.zip
Uploading the intermediate file: upload_file_15.zip
  14:46:24 STEP 1537 H= 384:15 +CPU= 24.338
  14:46:49 STEP 1538 H= 384:30 +CPU= 24.355
  14:47:13 STEP 1539 H= 384:45 +CPU= 24.347
  14:47:38 STEP 1540 H= 385:00 +CPU= 24.846
  14:48:03 STEP 1541 H= 385:15 +CPU= 24.530
  14:48:27 STEP 1542 H= 385:30 +CPU= 24.745
  14:48:52 STEP 1543 H= 385:45 +CPU= 24.543
  14:49:17 STEP 1544 H= 386:00 +CPU= 24.716
  14:49:42 STEP 1545 H= 386:15 +CPU= 24.629
  14:50:06 STEP 1546 H= 386:30 +CPU= 24.373
  14:50:31 STEP 1547 H= 386:45 +CPU= 24.565
  14:51:01 STEP 1548 H= 387:00 +CPU= 29.840
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001548
  14:51:25 STEP 1549 H= 387:15 +CPU= 24.358
  14:51:50 STEP 1550 H= 387:30 +CPU= 24.524
  14:52:14 STEP 1551 H= 387:45 +CPU= 24.182
  14:52:39 STEP 1552 H= 388:00 +CPU= 24.704
  14:53:03 STEP 1553 H= 388:15 +CPU= 24.370
  14:53:28 STEP 1554 H= 388:30 +CPU= 24.551
  14:53:53 STEP 1555 H= 388:45 +CPU= 24.527
  14:54:18 STEP 1556 H= 389:00 +CPU= 25.097
  14:54:42 STEP 1557 H= 389:15 +CPU= 24.529
  14:55:07 STEP 1558 H= 389:30 +CPU= 24.540
  14:55:32 STEP 1559 H= 389:45 +CPU= 24.515
  14:56:02 STEP 1560 H= 390:00 +CPU= 30.035
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001560
  14:56:27 STEP 1561 H= 390:15 +CPU= 24.628
  14:56:51 STEP 1562 H= 390:30 +CPU= 24.400
  14:57:16 STEP 1563 H= 390:45 +CPU= 24.617
  14:57:41 STEP 1564 H= 391:00 +CPU= 24.958
  14:58:05 STEP 1565 H= 391:15 +CPU= 24.442
  14:58:30 STEP 1566 H= 391:30 +CPU= 24.424
  14:58:54 STEP 1567 H= 391:45 +CPU= 24.421
  14:59:19 STEP 1568 H= 392:00 +CPU= 24.650
  14:59:44 STEP 1569 H= 392:15 +CPU= 24.503
  15:00:08 STEP 1570 H= 392:30 +CPU= 24.424
  15:00:33 STEP 1571 H= 392:45 +CPU= 24.350
  15:01:03 STEP 1572 H= 393:00 +CPU= 30.020
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001572
  15:01:27 STEP 1573 H= 393:15 +CPU= 24.467
  15:01:52 STEP 1574 H= 393:30 +CPU= 24.524
  15:02:17 STEP 1575 H= 393:45 +CPU= 24.559
  15:02:42 STEP 1576 H= 394:00 +CPU= 24.986
  15:03:07 STEP 1577 H= 394:15 +CPU= 25.166
  15:03:32 STEP 1578 H= 394:30 +CPU= 24.863
  15:03:57 STEP 1579 H= 394:45 +CPU= 24.728
  15:04:22 STEP 1580 H= 395:00 +CPU= 25.186
  15:04:47 STEP 1581 H= 395:15 +CPU= 24.693
  15:05:12 STEP 1582 H= 395:30 +CPU= 24.656
  15:05:37 STEP 1583 H= 395:45 +CPU= 24.570
  15:06:11 STEP 1584 H= 396:00 +CPU= 34.021
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001584
  15:06:36 STEP 1585 H= 396:15 +CPU= 24.655
  15:07:01 STEP 1586 H= 396:30 +CPU= 24.809
  15:07:25 STEP 1587 H= 396:45 +CPU= 24.365
  15:07:50 STEP 1588 H= 397:00 +CPU= 25.014
  15:08:15 STEP 1589 H= 397:15 +CPU= 24.742
  15:08:40 STEP 1590 H= 397:30 +CPU= 24.593
  15:09:04 STEP 1591 H= 397:45 +CPU= 24.368
  15:09:29 STEP 1592 H= 398:00 +CPU= 24.807
  15:09:54 STEP 1593 H= 398:15 +CPU= 24.435
  15:10:19 STEP 1594 H= 398:30 +CPU= 24.681
  15:10:43 STEP 1595 H= 398:45 +CPU= 24.512
  15:11:14 STEP 1596 H= 399:00 +CPU= 30.211
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001596
  15:11:39 STEP 1597 H= 399:15 +CPU= 24.853
  15:12:03 STEP 1598 H= 399:30 +CPU= 24.608
  15:12:28 STEP 1599 H= 399:45 +CPU= 24.534
  15:12:53 STEP 1600 H= 400:00 +CPU= 25.067
  15:13:18 STEP 1601 H= 400:15 +CPU= 24.564
  15:13:43 STEP 1602 H= 400:30 +CPU= 24.574
  15:14:07 STEP 1603 H= 400:45 +CPU= 24.787
  15:14:33 STEP 1604 H= 401:00 +CPU= 25.164
  15:14:58 STEP 1605 H= 401:15 +CPU= 24.698
  15:15:22 STEP 1606 H= 401:30 +CPU= 24.628
  15:15:47 STEP 1607 H= 401:45 +CPU= 24.630
  15:16:18 STEP 1608 H= 402:00 +CPU= 30.305
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001608
  15:16:42 STEP 1609 H= 402:15 +CPU= 24.451
  15:17:07 STEP 1610 H= 402:30 +CPU= 24.636
  15:17:31 STEP 1611 H= 402:45 +CPU= 24.416
  15:17:57 STEP 1612 H= 403:00 +CPU= 25.023
  15:18:21 STEP 1613 H= 403:15 +CPU= 24.520
  15:18:46 STEP 1614 H= 403:30 +CPU= 24.659
  15:19:11 STEP 1615 H= 403:45 +CPU= 24.632
  15:19:36 STEP 1616 H= 404:00 +CPU= 24.902
  15:20:00 STEP 1617 H= 404:15 +CPU= 24.346
  15:20:25 STEP 1618 H= 404:30 +CPU= 24.407
  15:20:49 STEP 1619 H= 404:45 +CPU= 24.449
  15:21:20 STEP 1620 H= 405:00 +CPU= 30.240
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001620
  15:21:45 STEP 1621 H= 405:15 +CPU= 24.865
  15:22:10 STEP 1622 H= 405:30 +CPU= 24.841
  15:22:35 STEP 1623 H= 405:45 +CPU= 24.702
  15:23:00 STEP 1624 H= 406:00 +CPU= 24.826
  15:23:24 STEP 1625 H= 406:15 +CPU= 24.520
  15:23:49 STEP 1626 H= 406:30 +CPU= 24.611
  15:24:14 STEP 1627 H= 406:45 +CPU= 24.815
  15:24:39 STEP 1628 H= 407:00 +CPU= 25.069
  15:25:04 STEP 1629 H= 407:15 +CPU= 24.836
  15:25:29 STEP 1630 H= 407:30 +CPU= 24.811
  15:25:54 STEP 1631 H= 407:45 +CPU= 24.653
  15:26:28 STEP 1632 H= 408:00 +CPU= 33.841
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12292480/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0_r193802484_16.zip
Uploading the intermediate file: upload_file_16.zip
  15:26:52 STEP 1633 H= 408:15 +CPU= 24.344
  15:27:17 STEP 1634 H= 408:30 +CPU= 24.626
  15:27:42 STEP 1635 H= 408:45 +CPU= 24.548
  15:28:07 STEP 1636 H= 409:00 +CPU= 24.957
  15:28:31 STEP 1637 H= 409:15 +CPU= 24.399
  15:28:56 STEP 1638 H= 409:30 +CPU= 24.507
  15:29:21 STEP 1639 H= 409:45 +CPU= 24.462
  15:29:46 STEP 1640 H= 410:00 +CPU= 24.944
  15:30:10 STEP 1641 H= 410:15 +CPU= 24.667
  15:30:35 STEP 1642 H= 410:30 +CPU= 24.841
  15:31:00 STEP 1643 H= 410:45 +CPU= 24.602
  15:31:31 STEP 1644 H= 411:00 +CPU= 30.377
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAh7zg+001644
  15:31:55 STEP 1645 H= 411:15 +CPU= 24.456
  15:32:20 STEP 1646 H= 411:30 +CPU= 24.402
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.641] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.642] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153236:1718317956:40553.643] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' : sigcontextptr=0x7ffef4b18450
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.643] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' : sigcontextptr=0x7ffef4b18360
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/7/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.672] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:Rig-38:1:1:141890:141890] [20240613:153237:1718317957:40554.672] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.423925444291031E-05 0.000000000000000E+00 0.691355267781127E-02
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.206526966684012E-04 0.000000000000000E+00 0.935652358382864E-02
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.454878732353724E-04 0.000000000000000E+00 0.201698410525029E-01
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.485243011900104E-05 0.000000000000000E+00 0.934060525935018E-02
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.213183018905672E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.423882183260994E-05 0.000000000000000E+00 0.815728254126922E-02
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.208449788460991E-04 0.000000000000000E+00 0.864807088376541E-02
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.213131169130038E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.162241732665113E-04 0.000000000000000E+00 0.308946119889553E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.953523581441828E-05 0.000000000000000E+00 0.517062609129729E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.244840695909499E-05 0.000000000000000E+00 0.114507124920694E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.259993987159177E-06 0.000000000000000E+00 0.677969714838815E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.182783209090017E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1644 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1644 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001644 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001644
 MPI-TASK:   1 -    304710820 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0077( 2.6%)
 NSTEP =  1644 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001644 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001644
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0060 SECONDS117899.9 Mbytes/s, TOTAL TIME=    0.1940( 3.1%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001644 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001644
 MPI-TASK:   1 -   4875373120 BYTES IN    125 RECORDS TRANSFERRED IN    0.0041 SECONDS******** Mbytes/s, TOTAL TIME=    0.1825( 2.2%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001644 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001644
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0034( 2.9%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001644 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001644
 MPI-TASK:   1 -    296131091 BYTES IN      4 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0054( 1.9%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1644 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1644 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001644 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001644
 MPI-TASK:   1 -    304710820 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS******** Mbytes/s, TOTAL TIME=    0.0087( 3.4%)
 15:31:31 STEP 1644 H= 411:00 +CPU= 30.377
ISTEP=    1645 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1645 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1645 STEPO   0AAA00AAA
 15:31:55 STEP 1645 H= 411:15 +CPU= 24.456
 NSTEP =  1646 STEPO   0AAA00AAA
 15:32:20 STEP 1646 H= 411:30 +CPU= 24.402
 NSTEP =  1647 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 15:31:02 FULLPOS-B DYNFPOS   1644     1.234    1.234    1.247    671:40    674:19 0.27488930426395E-04       2GB       0MB
 15:31:04 FULLPOS-V DYNFPOS   1644     2.522    2.522    2.544    671:43    674:21 0.27488930426395E-04       2GB       0MB
 15:31:05 FULLPOS-T DYNFPOS   1644     0.522    0.522    0.525    671:43    674:22 0.27488930426395E-04       2GB       0MB
 15:31:05 0AAA00AAA STEPO     1644     0.529    0.529    0.532    671:44    674:22 0.27488930426395E-04       2GB       0MB
 15:31:31 0AAA00AAA STEPO     1645    25.228   25.228   25.365    672:09    674:48 0.27488930426395E-04       2GB       0MB
 15:31:55 0AAA00AAA STEPO     1646    24.456   24.456   24.542    672:34    675:12 0.27488930426395E-04       2GB       0MB
 15:32:20 0AAA00AAA STEPO     1647    24.402   24.402   24.482    672:58    675:37 0.27488930426395E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1632",
 CTIME   = "00170000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1620, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/7/progress_file_12292480.xml

<running_values>
  <last_cpu_time>40396.770000</last_cpu_time>
  <upload_file_number>17</upload_file_number>
  <last_iter>1647</last_iter>
  <last_upload>1468800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
13 Jun 2024 18:22:08 1539347 22446086 oifs_43r3_bl_a37d_2016092300_20_1018_12292480_0 864,000 23,172 0.0268


©2024 climateprediction.net