climateprediction.net home page
Task 22449011

Task 22449011

Name oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2
Workunit 12290125
Created 15 Jun 2024, 5:53:40 UTC
Sent 15 Jun 2024, 5:53:45 UTC
Report deadline 14 Aug 2024, 5:53:45 UTC
Received 15 Jun 2024, 12:53:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1520999
Run time 5 hours 53 min 10 sec
CPU time 5 hours 51 min 13 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 8.51 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.34 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,825.71 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
:26 STEP 1479 H= 369:45 +CPU= 10.936
  12:44:38 STEP 1480 H= 370:00 +CPU= 11.262
  12:44:49 STEP 1481 H= 370:15 +CPU= 10.930
  12:45:00 STEP 1482 H= 370:30 +CPU= 11.110
  12:45:11 STEP 1483 H= 370:45 +CPU= 10.954
  12:45:22 STEP 1484 H= 371:00 +CPU= 11.193
  12:45:33 STEP 1485 H= 371:15 +CPU= 10.894
  12:45:44 STEP 1486 H= 371:30 +CPU= 10.872
  12:45:55 STEP 1487 H= 371:45 +CPU= 10.874
  12:46:11 STEP 1488 H= 372:00 +CPU= 15.792
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001488
  12:46:22 STEP 1489 H= 372:15 +CPU= 10.950
  12:46:32 STEP 1490 H= 372:30 +CPU= 10.890
  12:46:43 STEP 1491 H= 372:45 +CPU= 10.909
  12:46:55 STEP 1492 H= 373:00 +CPU= 11.287
  12:47:06 STEP 1493 H= 373:15 +CPU= 10.834
  12:47:16 STEP 1494 H= 373:30 +CPU= 10.821
  12:47:27 STEP 1495 H= 373:45 +CPU= 10.944
  12:47:39 STEP 1496 H= 374:00 +CPU= 11.226
  12:47:50 STEP 1497 H= 374:15 +CPU= 10.965
  12:48:01 STEP 1498 H= 374:30 +CPU= 10.990
  12:48:12 STEP 1499 H= 374:45 +CPU= 11.184
  12:48:26 STEP 1500 H= 375:00 +CPU= 14.257
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001500
  12:48:37 STEP 1501 H= 375:15 +CPU= 10.905
  12:48:48 STEP 1502 H= 375:30 +CPU= 11.094
  12:48:59 STEP 1503 H= 375:45 +CPU= 10.980
  12:49:11 STEP 1504 H= 376:00 +CPU= 11.466
  12:49:22 STEP 1505 H= 376:15 +CPU= 10.959
  12:49:33 STEP 1506 H= 376:30 +CPU= 11.083
  12:49:44 STEP 1507 H= 376:45 +CPU= 10.963
  12:49:55 STEP 1508 H= 377:00 +CPU= 11.472
  12:50:06 STEP 1509 H= 377:15 +CPU= 10.996
  12:50:17 STEP 1510 H= 377:30 +CPU= 10.912
  12:50:28 STEP 1511 H= 377:45 +CPU= 10.703
  12:50:42 STEP 1512 H= 378:00 +CPU= 13.924
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001512
  12:50:53 STEP 1513 H= 378:15 +CPU= 10.788
  12:51:04 STEP 1514 H= 378:30 +CPU= 10.859
  12:51:15 STEP 1515 H= 378:45 +CPU= 10.883
  12:51:26 STEP 1516 H= 379:00 +CPU= 11.074
  12:51:37 STEP 1517 H= 379:15 +CPU= 10.893
  12:51:47 STEP 1518 H= 379:30 +CPU= 10.822
  12:51:58 STEP 1519 H= 379:45 +CPU= 10.883
  12:52:09 STEP 1520 H= 380:00 +CPU= 11.080
  12:52:20 STEP 1521 H= 380:15 +CPU= 10.479
  12:52:31 STEP 1522 H= 380:30 +CPU= 10.819
  12:52:42 STEP 1523 H= 380:45 +CPU= 10.851
  12:52:56 STEP 1524 H= 381:00 +CPU= 14.169
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001524
  12:53:07 STEP 1525 H= 381:15 +CPU= 10.910
  12:53:18 STEP 1526 H= 381:30 +CPU= 11.013
  12:53:29 STEP 1527 H= 381:45 +CPU= 10.857
  12:53:40 STEP 1528 H= 382:00 +CPU= 11.155
  12:53:51 STEP 1529 H= 382:15 +CPU= 10.939
  12:54:02 STEP 1530 H= 382:30 +CPU= 10.998
  12:54:13 STEP 1531 H= 382:45 +CPU= 10.914
  12:54:24 STEP 1532 H= 383:00 +CPU= 11.220
  12:54:35 STEP 1533 H= 383:15 +CPU= 10.826
  12:54:46 STEP 1534 H= 383:30 +CPU= 10.898
  12:54:57 STEP 1535 H= 383:45 +CPU= 10.829
  12:55:13 STEP 1536 H= 384:00 +CPU= 16.032
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2_r1715563272_15.zip
  12:55:24 STEP 1537 H= 384:15 +CPU= 11.166
Uploading the intermediate file: upload_file_15.zip
  12:55:35 STEP 1538 H= 384:30 +CPU= 11.139
  12:55:46 STEP 1539 H= 384:45 +CPU= 10.929
  12:55:57 STEP 1540 H= 385:00 +CPU= 11.209
  12:56:09 STEP 1541 H= 385:15 +CPU= 10.967
  12:56:20 STEP 1542 H= 385:30 +CPU= 11.034
  12:56:30 STEP 1543 H= 385:45 +CPU= 10.854
  12:56:42 STEP 1544 H= 386:00 +CPU= 11.442
  12:56:53 STEP 1545 H= 386:15 +CPU= 11.182
  12:57:04 STEP 1546 H= 386:30 +CPU= 10.806
  12:57:15 STEP 1547 H= 386:45 +CPU= 10.724
  12:57:29 STEP 1548 H= 387:00 +CPU= 13.963
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001548
  12:57:40 STEP 1549 H= 387:15 +CPU= 10.953
  12:57:51 STEP 1550 H= 387:30 +CPU= 11.047
  12:58:02 STEP 1551 H= 387:45 +CPU= 10.941
  12:58:13 STEP 1552 H= 388:00 +CPU= 11.250
  12:58:24 STEP 1553 H= 388:15 +CPU= 10.879
  12:58:35 STEP 1554 H= 388:30 +CPU= 10.989
  12:58:46 STEP 1555 H= 388:45 +CPU= 11.108
  12:58:58 STEP 1556 H= 389:00 +CPU= 11.451
  12:59:09 STEP 1557 H= 389:15 +CPU= 10.860
  12:59:20 STEP 1558 H= 389:30 +CPU= 10.938
  12:59:31 STEP 1559 H= 389:45 +CPU= 10.904
  12:59:45 STEP 1560 H= 390:00 +CPU= 14.226
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001560
  12:59:56 STEP 1561 H= 390:15 +CPU= 10.910
  13:00:07 STEP 1562 H= 390:30 +CPU= 10.914
  13:00:18 STEP 1563 H= 390:45 +CPU= 10.928
  13:00:29 STEP 1564 H= 391:00 +CPU= 11.432
  13:00:40 STEP 1565 H= 391:15 +CPU= 10.961
  13:00:51 STEP 1566 H= 391:30 +CPU= 10.879
  13:01:02 STEP 1567 H= 391:45 +CPU= 10.874
  13:01:13 STEP 1568 H= 392:00 +CPU= 11.417
  13:01:24 STEP 1569 H= 392:15 +CPU= 11.065
  13:01:35 STEP 1570 H= 392:30 +CPU= 10.951
  13:01:46 STEP 1571 H= 392:45 +CPU= 10.983
  13:02:01 STEP 1572 H= 393:00 +CPU= 14.286
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001572
  13:02:12 STEP 1573 H= 393:15 +CPU= 10.939
  13:02:23 STEP 1574 H= 393:30 +CPU= 10.905
  13:02:34 STEP 1575 H= 393:45 +CPU= 10.871
  13:02:45 STEP 1576 H= 394:00 +CPU= 11.274
  13:02:56 STEP 1577 H= 394:15 +CPU= 10.883
  13:03:07 STEP 1578 H= 394:30 +CPU= 11.014
  13:03:18 STEP 1579 H= 394:45 +CPU= 10.914
  13:03:29 STEP 1580 H= 395:00 +CPU= 11.275
  13:03:40 STEP 1581 H= 395:15 +CPU= 11.033
  13:03:51 STEP 1582 H= 395:30 +CPU= 10.878
  13:04:02 STEP 1583 H= 395:45 +CPU= 11.235
  13:04:19 STEP 1584 H= 396:00 +CPU= 16.150
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001584
  13:04:30 STEP 1585 H= 396:15 +CPU= 10.982
  13:04:41 STEP 1586 H= 396:30 +CPU= 11.149
  13:04:52 STEP 1587 H= 396:45 +CPU= 11.044
  13:05:03 STEP 1588 H= 397:00 +CPU= 11.449
  13:05:14 STEP 1589 H= 397:15 +CPU= 11.034
  13:05:25 STEP 1590 H= 397:30 +CPU= 11.033
  13:05:36 STEP 1591 H= 397:45 +CPU= 10.811
  13:05:47 STEP 1592 H= 398:00 +CPU= 11.185
  13:05:58 STEP 1593 H= 398:15 +CPU= 10.752
  13:06:09 STEP 1594 H= 398:30 +CPU= 10.615
  13:06:19 STEP 1595 H= 398:45 +CPU= 10.684
  13:06:33 STEP 1596 H= 399:00 +CPU= 13.875
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001596
  13:06:44 STEP 1597 H= 399:15 +CPU= 10.618
  13:06:55 STEP 1598 H= 399:30 +CPU= 10.631
  13:07:06 STEP 1599 H= 399:45 +CPU= 10.841
  13:07:17 STEP 1600 H= 400:00 +CPU= 11.102
  13:07:27 STEP 1601 H= 400:15 +CPU= 10.661
  13:07:38 STEP 1602 H= 400:30 +CPU= 10.678
  13:07:49 STEP 1603 H= 400:45 +CPU= 10.716
  13:08:00 STEP 1604 H= 401:00 +CPU= 11.064
  13:08:11 STEP 1605 H= 401:15 +CPU= 10.644
  13:08:21 STEP 1606 H= 401:30 +CPU= 10.811
  13:08:32 STEP 1607 H= 401:45 +CPU= 10.657
  13:08:46 STEP 1608 H= 402:00 +CPU= 13.953
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001608
  13:08:57 STEP 1609 H= 402:15 +CPU= 10.645
  13:09:07 STEP 1610 H= 402:30 +CPU= 10.729
  13:09:18 STEP 1611 H= 402:45 +CPU= 10.570
  13:09:29 STEP 1612 H= 403:00 +CPU= 11.207
  13:09:40 STEP 1613 H= 403:15 +CPU= 10.663
  13:09:51 STEP 1614 H= 403:30 +CPU= 10.818
  13:10:01 STEP 1615 H= 403:45 +CPU= 10.658
  13:10:13 STEP 1616 H= 404:00 +CPU= 11.042
  13:10:23 STEP 1617 H= 404:15 +CPU= 10.714
  13:10:34 STEP 1618 H= 404:30 +CPU= 10.691
  13:10:45 STEP 1619 H= 404:45 +CPU= 10.748
  13:10:59 STEP 1620 H= 405:00 +CPU= 13.863
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001620
  13:11:09 STEP 1621 H= 405:15 +CPU= 10.655
  13:11:20 STEP 1622 H= 405:30 +CPU= 10.611
  13:11:31 STEP 1623 H= 405:45 +CPU= 10.743
  13:11:42 STEP 1624 H= 406:00 +CPU= 11.040
  13:11:52 STEP 1625 H= 406:15 +CPU= 10.679
  13:12:03 STEP 1626 H= 406:30 +CPU= 10.763
  13:12:14 STEP 1627 H= 406:45 +CPU= 10.610
  13:12:25 STEP 1628 H= 407:00 +CPU= 10.991
  13:12:36 STEP 1629 H= 407:15 +CPU= 10.710
  13:12:46 STEP 1630 H= 407:30 +CPU= 10.750
  13:12:57 STEP 1631 H= 407:45 +CPU= 10.685
  13:13:13 STEP 1632 H= 408:00 +CPU= 15.568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2_r1715563272_16.zip
Uploading the intermediate file: upload_file_16.zip
  13:13:23 STEP 1633 H= 408:15 +CPU= 10.730
  13:13:34 STEP 1634 H= 408:30 +CPU= 10.890
  13:13:45 STEP 1635 H= 408:45 +CPU= 10.734
  13:13:56 STEP 1636 H= 409:00 +CPU= 11.187
  13:14:07 STEP 1637 H= 409:15 +CPU= 10.841
  13:14:18 STEP 1638 H= 409:30 +CPU= 10.782
  13:14:29 STEP 1639 H= 409:45 +CPU= 10.869
  13:14:40 STEP 1640 H= 410:00 +CPU= 11.352
  13:14:51 STEP 1641 H= 410:15 +CPU= 10.663
  13:15:02 STEP 1642 H= 410:30 +CPU= 10.693
  13:15:12 STEP 1643 H= 410:45 +CPU= 10.723
  13:15:26 STEP 1644 H= 411:00 +CPU= 13.916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001644
  13:15:37 STEP 1645 H= 411:15 +CPU= 10.811
  13:15:48 STEP 1646 H= 411:30 +CPU= 10.754
  13:15:59 STEP 1647 H= 411:45 +CPU= 10.724
  13:16:10 STEP 1648 H= 412:00 +CPU= 11.076
  13:16:21 STEP 1649 H= 412:15 +CPU= 10.666
  13:16:31 STEP 1650 H= 412:30 +CPU= 10.715
  13:16:42 STEP 1651 H= 412:45 +CPU= 10.682
  13:16:53 STEP 1652 H= 413:00 +CPU= 11.155
  13:17:04 STEP 1653 H= 413:15 +CPU= 10.670
  13:17:15 STEP 1654 H= 413:30 +CPU= 10.857
  13:17:26 STEP 1655 H= 413:45 +CPU= 10.756
  13:17:39 STEP 1656 H= 414:00 +CPU= 13.927
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001656
  13:17:50 STEP 1657 H= 414:15 +CPU= 10.754
  13:18:01 STEP 1658 H= 414:30 +CPU= 10.747
  13:18:12 STEP 1659 H= 414:45 +CPU= 10.802
  13:18:23 STEP 1660 H= 415:00 +CPU= 11.163
  13:18:34 STEP 1661 H= 415:15 +CPU= 10.804
  13:18:45 STEP 1662 H= 415:30 +CPU= 10.867
  13:18:56 STEP 1663 H= 415:45 +CPU= 10.771
  13:19:07 STEP 1664 H= 416:00 +CPU= 11.144
  13:19:17 STEP 1665 H= 416:15 +CPU= 10.677
  13:19:28 STEP 1666 H= 416:30 +CPU= 10.753
  13:19:39 STEP 1667 H= 416:45 +CPU= 10.695
  13:19:53 STEP 1668 H= 417:00 +CPU= 13.896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001668
  13:20:04 STEP 1669 H= 417:15 +CPU= 10.782
  13:20:14 STEP 1670 H= 417:30 +CPU= 10.800
  13:20:25 STEP 1671 H= 417:45 +CPU= 10.601
  13:20:36 STEP 1672 H= 418:00 +CPU= 11.062
  13:20:47 STEP 1673 H= 418:15 +CPU= 10.658
  13:20:58 STEP 1674 H= 418:30 +CPU= 10.694
  13:21:08 STEP 1675 H= 418:45 +CPU= 10.766
  13:21:19 STEP 1676 H= 419:00 +CPU= 11.011
  13:21:30 STEP 1677 H= 419:15 +CPU= 10.695
  13:21:41 STEP 1678 H= 419:30 +CPU= 10.800
  13:21:52 STEP 1679 H= 419:45 +CPU= 10.790
  13:22:07 STEP 1680 H= 420:00 +CPU= 15.697
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001680
  13:22:18 STEP 1681 H= 420:15 +CPU= 10.626
  13:22:29 STEP 1682 H= 420:30 +CPU= 10.710
  13:22:39 STEP 1683 H= 420:45 +CPU= 10.591
  13:22:50 STEP 1684 H= 421:00 +CPU= 10.980
  13:23:01 STEP 1685 H= 421:15 +CPU= 10.710
  13:23:12 STEP 1686 H= 421:30 +CPU= 10.665
  13:23:23 STEP 1687 H= 421:45 +CPU= 10.746
  13:23:34 STEP 1688 H= 422:00 +CPU= 11.038
  13:23:44 STEP 1689 H= 422:15 +CPU= 10.627
  13:23:55 STEP 1690 H= 422:30 +CPU= 10.695
  13:24:06 STEP 1691 H= 422:45 +CPU= 10.714
  13:24:20 STEP 1692 H= 423:00 +CPU= 13.904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001692
  13:24:30 STEP 1693 H= 423:15 +CPU= 10.686
  13:24:41 STEP 1694 H= 423:30 +CPU= 10.653
  13:24:52 STEP 1695 H= 423:45 +CPU= 10.453
  13:25:03 STEP 1696 H= 424:00 +CPU= 10.974
  13:25:13 STEP 1697 H= 424:15 +CPU= 10.545
  13:25:24 STEP 1698 H= 424:30 +CPU= 10.601
  13:25:34 STEP 1699 H= 424:45 +CPU= 10.672
  13:25:45 STEP 1700 H= 425:00 +CPU= 11.042
  13:25:56 STEP 1701 H= 425:15 +CPU= 10.679
  13:26:07 STEP 1702 H= 425:30 +CPU= 10.674
  13:26:18 STEP 1703 H= 425:45 +CPU= 10.714
  13:26:31 STEP 1704 H= 426:00 +CPU= 13.874
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001704
  13:26:42 STEP 1705 H= 426:15 +CPU= 10.592
  13:26:53 STEP 1706 H= 426:30 +CPU= 10.662
  13:27:03 STEP 1707 H= 426:45 +CPU= 10.628
  13:27:14 STEP 1708 H= 427:00 +CPU= 10.988
  13:27:25 STEP 1709 H= 427:15 +CPU= 10.688
  13:27:36 STEP 1710 H= 427:30 +CPU= 10.643
  13:27:46 STEP 1711 H= 427:45 +CPU= 10.641
  13:27:58 STEP 1712 H= 428:00 +CPU= 11.041
  13:28:08 STEP 1713 H= 428:15 +CPU= 10.698
  13:28:19 STEP 1714 H= 428:30 +CPU= 10.469
  13:28:29 STEP 1715 H= 428:45 +CPU= 10.662
  13:28:43 STEP 1716 H= 429:00 +CPU= 13.863
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001716
  13:28:54 STEP 1717 H= 429:15 +CPU= 10.622
  13:29:05 STEP 1718 H= 429:30 +CPU= 10.749
  13:29:15 STEP 1719 H= 429:45 +CPU= 10.751
  13:29:27 STEP 1720 H= 430:00 +CPU= 11.046
  13:29:37 STEP 1721 H= 430:15 +CPU= 10.657
  13:29:48 STEP 1722 H= 430:30 +CPU= 10.646
  13:29:58 STEP 1723 H= 430:45 +CPU= 10.561
  13:30:10 STEP 1724 H= 431:00 +CPU= 11.041
  13:30:20 STEP 1725 H= 431:15 +CPU= 10.604
  13:30:31 STEP 1726 H= 431:30 +CPU= 10.558
  13:30:41 STEP 1727 H= 431:45 +CPU= 10.629
  13:30:57 STEP 1728 H= 432:00 +CPU= 15.466
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2_r1715563272_17.zip
  13:31:08 STEP 1729 H= 432:15 +CPU= 10.751
Uploading the intermediate file: upload_file_17.zip
  13:31:18 STEP 1730 H= 432:30 +CPU= 10.773
  13:31:29 STEP 1731 H= 432:45 +CPU= 10.535
  13:31:40 STEP 1732 H= 433:00 +CPU= 11.160
  13:31:51 STEP 1733 H= 433:15 +CPU= 10.984
  13:32:02 STEP 1734 H= 433:30 +CPU= 10.831
  13:32:13 STEP 1735 H= 433:45 +CPU= 10.650
  13:32:24 STEP 1736 H= 434:00 +CPU= 11.085
  13:32:35 STEP 1737 H= 434:15 +CPU= 10.624
  13:32:45 STEP 1738 H= 434:30 +CPU= 10.647
  13:32:56 STEP 1739 H= 434:45 +CPU= 10.621
  13:33:10 STEP 1740 H= 435:00 +CPU= 13.841
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001740
  13:33:20 STEP 1741 H= 435:15 +CPU= 10.657
  13:33:31 STEP 1742 H= 435:30 +CPU= 10.705
  13:33:42 STEP 1743 H= 435:45 +CPU= 10.625
  13:33:53 STEP 1744 H= 436:00 +CPU= 10.994
  13:34:03 STEP 1745 H= 436:15 +CPU= 10.618
  13:34:14 STEP 1746 H= 436:30 +CPU= 10.667
  13:34:25 STEP 1747 H= 436:45 +CPU= 10.685
  13:34:36 STEP 1748 H= 437:00 +CPU= 11.089
  13:34:47 STEP 1749 H= 437:15 +CPU= 10.694
  13:34:57 STEP 1750 H= 437:30 +CPU= 10.689
  13:35:08 STEP 1751 H= 437:45 +CPU= 10.767
  13:35:22 STEP 1752 H= 438:00 +CPU= 13.989
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001752
  13:35:33 STEP 1753 H= 438:15 +CPU= 10.715
  13:35:44 STEP 1754 H= 438:30 +CPU= 10.778
  13:35:54 STEP 1755 H= 438:45 +CPU= 10.640
  13:36:06 STEP 1756 H= 439:00 +CPU= 11.214
  13:36:16 STEP 1757 H= 439:15 +CPU= 10.742
  13:36:27 STEP 1758 H= 439:30 +CPU= 10.705
  13:36:38 STEP 1759 H= 439:45 +CPU= 10.752
  13:36:49 STEP 1760 H= 440:00 +CPU= 11.165
  13:37:00 STEP 1761 H= 440:15 +CPU= 10.805
  13:37:11 STEP 1762 H= 440:30 +CPU= 10.704
  13:37:21 STEP 1763 H= 440:45 +CPU= 10.719
  13:37:35 STEP 1764 H= 441:00 +CPU= 13.957
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001764
  13:37:46 STEP 1765 H= 441:15 +CPU= 10.789
  13:37:57 STEP 1766 H= 441:30 +CPU= 10.701
  13:38:08 STEP 1767 H= 441:45 +CPU= 10.804
  13:38:19 STEP 1768 H= 442:00 +CPU= 11.097
  13:38:30 STEP 1769 H= 442:15 +CPU= 10.767
  13:38:40 STEP 1770 H= 442:30 +CPU= 10.751
  13:38:51 STEP 1771 H= 442:45 +CPU= 10.722
  13:39:02 STEP 1772 H= 443:00 +CPU= 11.115
  13:39:13 STEP 1773 H= 443:15 +CPU= 10.922
  13:39:24 STEP 1774 H= 443:30 +CPU= 10.848
  13:39:35 STEP 1775 H= 443:45 +CPU= 10.799
  13:39:51 STEP 1776 H= 444:00 +CPU= 15.648
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001776
  13:40:01 STEP 1777 H= 444:15 +CPU= 10.712
  13:40:12 STEP 1778 H= 444:30 +CPU= 10.702
  13:40:23 STEP 1779 H= 444:45 +CPU= 10.690
  13:40:34 STEP 1780 H= 445:00 +CPU= 11.107
  13:40:45 STEP 1781 H= 445:15 +CPU= 10.672
  13:40:55 STEP 1782 H= 445:30 +CPU= 10.842
  13:41:06 STEP 1783 H= 445:45 +CPU= 10.737
  13:41:17 STEP 1784 H= 446:00 +CPU= 11.021
  13:41:28 STEP 1785 H= 446:15 +CPU= 10.771
  13:41:39 STEP 1786 H= 446:30 +CPU= 10.669
  13:41:49 STEP 1787 H= 446:45 +CPU= 10.724
  13:42:03 STEP 1788 H= 447:00 +CPU= 13.878
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001788
  13:42:14 STEP 1789 H= 447:15 +CPU= 10.782
  13:42:25 STEP 1790 H= 447:30 +CPU= 10.809
  13:42:36 STEP 1791 H= 447:45 +CPU= 10.667
  13:42:47 STEP 1792 H= 448:00 +CPU= 11.150
  13:42:58 STEP 1793 H= 448:15 +CPU= 10.769
  13:43:08 STEP 1794 H= 448:30 +CPU= 10.774
  13:43:19 STEP 1795 H= 448:45 +CPU= 10.680
  13:43:30 STEP 1796 H= 449:00 +CPU= 11.131
  13:43:41 STEP 1797 H= 449:15 +CPU= 10.653
  13:43:52 STEP 1798 H= 449:30 +CPU= 10.722
  13:44:02 STEP 1799 H= 449:45 +CPU= 10.707
  13:44:16 STEP 1800 H= 450:00 +CPU= 14.005
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001800
  13:44:27 STEP 1801 H= 450:15 +CPU= 10.817
  13:44:38 STEP 1802 H= 450:30 +CPU= 10.804
  13:44:49 STEP 1803 H= 450:45 +CPU= 10.800
  13:45:00 STEP 1804 H= 451:00 +CPU= 11.101
  13:45:11 STEP 1805 H= 451:15 +CPU= 10.840
  13:45:22 STEP 1806 H= 451:30 +CPU= 10.725
  13:45:32 STEP 1807 H= 451:45 +CPU= 10.596
  13:45:43 STEP 1808 H= 452:00 +CPU= 11.101
  13:45:54 STEP 1809 H= 452:15 +CPU= 10.774
  13:46:05 STEP 1810 H= 452:30 +CPU= 10.808
  13:46:16 STEP 1811 H= 452:45 +CPU= 10.726
  13:46:30 STEP 1812 H= 453:00 +CPU= 13.994
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001812
  13:46:40 STEP 1813 H= 453:15 +CPU= 10.639
  13:46:51 STEP 1814 H= 453:30 +CPU= 10.753
  13:47:02 STEP 1815 H= 453:45 +CPU= 10.685
  13:47:13 STEP 1816 H= 454:00 +CPU= 11.024
  13:47:24 STEP 1817 H= 454:15 +CPU= 10.712
  13:47:34 STEP 1818 H= 454:30 +CPU= 10.714
  13:47:45 STEP 1819 H= 454:45 +CPU= 10.715
  13:47:56 STEP 1820 H= 455:00 +CPU= 11.096
  13:48:07 STEP 1821 H= 455:15 +CPU= 10.733
  13:48:18 STEP 1822 H= 455:30 +CPU= 10.696
  13:48:28 STEP 1823 H= 455:45 +CPU= 10.679
  13:48:44 STEP 1824 H= 456:00 +CPU= 15.742
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290125/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2_r1715563272_18.zip
  13:48:55 STEP 1825 H= 456:15 +CPU= 10.873
Uploading the intermediate file: upload_file_18.zip
  13:49:06 STEP 1826 H= 456:30 +CPU= 10.765
  13:49:17 STEP 1827 H= 456:45 +CPU= 10.813
  13:49:28 STEP 1828 H= 457:00 +CPU= 11.181
  13:49:39 STEP 1829 H= 457:15 +CPU= 10.922
  13:49:50 STEP 1830 H= 457:30 +CPU= 10.809
  13:50:01 STEP 1831 H= 457:45 +CPU= 10.717
  13:50:12 STEP 1832 H= 458:00 +CPU= 11.250
  13:50:23 STEP 1833 H= 458:15 +CPU= 10.743
  13:50:33 STEP 1834 H= 458:30 +CPU= 10.758
  13:50:44 STEP 1835 H= 458:45 +CPU= 10.761
  13:50:58 STEP 1836 H= 459:00 +CPU= 13.962
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001836
  13:51:09 STEP 1837 H= 459:15 +CPU= 10.893
  13:51:20 STEP 1838 H= 459:30 +CPU= 10.844
  13:51:31 STEP 1839 H= 459:45 +CPU= 10.865
  13:51:42 STEP 1840 H= 460:00 +CPU= 11.241
  13:51:53 STEP 1841 H= 460:15 +CPU= 10.793
  13:52:04 STEP 1842 H= 460:30 +CPU= 10.866
  13:52:15 STEP 1843 H= 460:45 +CPU= 10.865
  13:52:26 STEP 1844 H= 461:00 +CPU= 11.139
  13:52:37 STEP 1845 H= 461:15 +CPU= 10.785
  13:52:47 STEP 1846 H= 461:30 +CPU= 10.810
  13:52:58 STEP 1847 H= 461:45 +CPU= 10.839
  13:53:12 STEP 1848 H= 462:00 +CPU= 13.944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001848
  13:53:23 STEP 1849 H= 462:15 +CPU= 10.789
  13:53:34 STEP 1850 H= 462:30 +CPU= 10.736
  13:53:45 STEP 1851 H= 462:45 +CPU= 10.798
  13:53:56 STEP 1852 H= 463:00 +CPU= 11.187
  13:54:07 STEP 1853 H= 463:15 +CPU= 10.779
  13:54:18 STEP 1854 H= 463:30 +CPU= 10.883
  13:54:28 STEP 1855 H= 463:45 +CPU= 10.822
  13:54:40 STEP 1856 H= 464:00 +CPU= 11.242
  13:54:51 STEP 1857 H= 464:15 +CPU= 10.817
  13:55:01 STEP 1858 H= 464:30 +CPU= 10.764
  13:55:12 STEP 1859 H= 464:45 +CPU= 10.840
  13:55:26 STEP 1860 H= 465:00 +CPU= 14.102
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001860
  13:55:37 STEP 1861 H= 465:15 +CPU= 10.882
  13:55:48 STEP 1862 H= 465:30 +CPU= 10.861
  13:55:59 STEP 1863 H= 465:45 +CPU= 10.813
  13:56:10 STEP 1864 H= 466:00 +CPU= 11.312
  13:56:21 STEP 1865 H= 466:15 +CPU= 10.802
  13:56:32 STEP 1866 H= 466:30 +CPU= 10.646
  13:56:43 STEP 1867 H= 466:45 +CPU= 10.881
  13:56:54 STEP 1868 H= 467:00 +CPU= 11.230
  13:57:05 STEP 1869 H= 467:15 +CPU= 10.806
  13:57:16 STEP 1870 H= 467:30 +CPU= 10.833
  13:57:26 STEP 1871 H= 467:45 +CPU= 10.859
  13:57:42 STEP 1872 H= 468:00 +CPU= 15.682
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001872
  13:57:53 STEP 1873 H= 468:15 +CPU= 10.903
  13:58:04 STEP 1874 H= 468:30 +CPU= 10.845
  13:58:15 STEP 1875 H= 468:45 +CPU= 10.902
  13:58:26 STEP 1876 H= 469:00 +CPU= 11.118
  13:58:37 STEP 1877 H= 469:15 +CPU= 10.805
  13:58:48 STEP 1878 H= 469:30 +CPU= 10.678
  13:58:58 STEP 1879 H= 469:45 +CPU= 10.779
  13:59:10 STEP 1880 H= 470:00 +CPU= 11.222
  13:59:20 STEP 1881 H= 470:15 +CPU= 10.748
  13:59:31 STEP 1882 H= 470:30 +CPU= 10.701
  13:59:42 STEP 1883 H= 470:45 +CPU= 10.760
  13:59:56 STEP 1884 H= 471:00 +CPU= 14.073
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001884
  14:00:07 STEP 1885 H= 471:15 +CPU= 10.779
  14:00:17 STEP 1886 H= 471:30 +CPU= 10.694
  14:00:28 STEP 1887 H= 471:45 +CPU= 10.753
  14:00:39 STEP 1888 H= 472:00 +CPU= 11.136
  14:00:50 STEP 1889 H= 472:15 +CPU= 10.812
  14:01:01 STEP 1890 H= 472:30 +CPU= 10.883
  14:01:12 STEP 1891 H= 472:45 +CPU= 10.818
  14:01:23 STEP 1892 H= 473:00 +CPU= 11.094
  14:01:34 STEP 1893 H= 473:15 +CPU= 10.858
  14:01:45 STEP 1894 H= 473:30 +CPU= 10.792
  14:01:56 STEP 1895 H= 473:45 +CPU= 10.767
  14:02:10 STEP 1896 H= 474:00 +CPU= 13.880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001896
  14:02:20 STEP 1897 H= 474:15 +CPU= 10.789
  14:02:31 STEP 1898 H= 474:30 +CPU= 10.768
  14:02:42 STEP 1899 H= 474:45 +CPU= 10.741
  14:02:53 STEP 1900 H= 475:00 +CPU= 11.125
  14:03:04 STEP 1901 H= 475:15 +CPU= 10.754
  14:03:15 STEP 1902 H= 475:30 +CPU= 10.823
  14:03:25 STEP 1903 H= 475:45 +CPU= 10.732
  14:03:36 STEP 1904 H= 476:00 +CPU= 11.056
  14:03:47 STEP 1905 H= 476:15 +CPU= 10.885
  14:03:58 STEP 1906 H= 476:30 +CPU= 10.716
  14:04:09 STEP 1907 H= 476:45 +CPU= 10.857
  14:04:23 STEP 1908 H= 477:00 +CPU= 13.944
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+001908
  14:04:34 STEP 1909 H= 477:15 +CPU= 10.761
  14:04:44 STEP 1910 H= 477:30 +CPU= 10.743
  14:04:55 STEP 1911 H= 477:45 +CPU= 10.766
  14:05:06 STEP 1912 H= 478:00 +CPU= 11.174
  14:05:17 STEP 1913 H= 478:15 +CPU= 10.846
  14:05:28 STEP 1914 H= 478:30 +CPU= 10.855
  14:05:39 STEP 1915 H= 478:45 +CPU= 10.765
  14:05:50 STEP 1916 H= 479:00 +CPU= 11.105
  14:06:01 STEP 1917 H= 479:15 +CPU= 10.805
  14:06:12 STEP 1918 H= 479:30 +CPU= 10.845
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.454] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140619:1718453179:21127.455] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' : sigcontextptr=0x7ffedaa22bd0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.455] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' : sigcontextptr=0x7ffedaa22ae0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/2/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.464] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:Linux-5600:1:1:432309:432309] [20240615:140620:1718453180:21128.464] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.660406790373394E-05 0.000000000000000E+00 0.579255144485780E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.255866730991742E-05 0.000000000000000E+00 0.890067453120761E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.403856386250256E-06 0.000000000000000E+00 0.536487947189341E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.173759279150913E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1908 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1908 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001908 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001908
 MPI-TASK:   1 -    319993460 BYTES IN      6 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0040( 2.5%)
 NSTEP =  1908 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001908 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001908
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0044 SECONDS160772.6 Mbytes/s, TOTAL TIME=    0.1145( 3.8%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001908 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001908
 MPI-TASK:   1 -   5119895360 BYTES IN    125 RECORDS TRANSFERRED IN    0.0030 SECONDS******** Mbytes/s, TOTAL TIME=    0.0954( 3.1%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001908 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001908
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0021( 4.8%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001908 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001908
 MPI-TASK:   1 -    311413731 BYTES IN      4 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0030( 3.3%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1908 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1908 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001908 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001908
 MPI-TASK:   1 -    319993460 BYTES IN      7 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0046( 4.3%)
 14:04:23 STEP 1908 H= 477:00 +CPU= 13.944
 NSTEP =  1909 STEPO   0AAA00AAA
 14:04:34 STEP 1909 H= 477:15 +CPU= 10.761
ISTEP=    1910 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1910 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1910 STEPO   0AAA00AAA
 14:04:44 STEP 1910 H= 477:30 +CPU= 10.743
 NSTEP =  1911 STEPO   0AAA00AAA
 14:04:55 STEP 1911 H= 477:45 +CPU= 10.766
 NSTEP =  1912 STEPO   0AAA00AAA
 14:05:06 STEP 1912 H= 478:00 +CPU= 11.174
 NSTEP =  1913 STEPO   0AAA00AAA
 14:05:17 STEP 1913 H= 478:15 +CPU= 10.846
 NSTEP =  1914 STEPO   0AAA00AAA
 14:05:28 STEP 1914 H= 478:30 +CPU= 10.855
ISTEP=    1915 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1915 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1915 STEPO   0AAA00AAA
 14:05:39 STEP 1915 H= 478:45 +CPU= 10.765
 NSTEP =  1916 STEPO   0AAA00AAA
 14:05:50 STEP 1916 H= 479:00 +CPU= 11.105
 NSTEP =  1917 STEPO   0AAA00AAA
 14:06:01 STEP 1917 H= 479:15 +CPU= 10.805
 NSTEP =  1918 STEPO   0AAA00AAA
 14:06:12 STEP 1918 H= 479:30 +CPU= 10.845
 NSTEP =  1919 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 14:05:06 0AAA00AAA STEPO     1913    11.174   11.174   11.198    350:00    350:55 0.24995288448096E-04       2GB       0MB
 14:05:17 0AAA00AAA STEPO     1914    10.846   10.846   10.863    350:11    351:06 0.24995288448096E-04       2GB       0MB
 14:05:28 0AAA00AAA STEPO     1915    10.855   10.855   10.878    350:22    351:16 0.24995288448096E-04       2GB       0MB
 14:05:39 0AAA00AAA STEPO     1916    10.765   10.765   10.782    350:33    351:27 0.24995288448096E-04       2GB       0MB
 14:05:50 0AAA00AAA STEPO     1917    11.105   11.105   11.123    350:44    351:38 0.24995288448096E-04       2GB       0MB
 14:06:01 0AAA00AAA STEPO     1918    10.806   10.806   10.830    350:55    351:49 0.24995288448096E-04       2GB       0MB
 14:06:12 0AAA00AAA STEPO     1919    10.845   10.845   10.863    351:05    352:00 0.24995288448096E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1872",
 CTIME   = "00191200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1860, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/2/progress_file_12290125.xml

<running_values>
  <last_cpu_time>21071.960000</last_cpu_time>
  <upload_file_number>19</upload_file_number>
  <last_iter>1919</last_iter>
  <last_upload>1641600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Jun 2024 09:08:47 1520999 22449011 oifs_43r3_bl_a1dy_2016092300_20_1018_12290125_2 864,000 10,362 0.0120


©2024 climateprediction.net