climateprediction.net home page
Task 22274922

Task 22274922

Name oifs_43r3_ps_0742_2003050100_123_972_12189386_0
Workunit 12189386
Created 23 Dec 2022, 6:08:34 UTC
Sent 16 Jan 2023, 11:21:52 UTC
Report deadline 15 Feb 2023, 11:21:52 UTC
Received 18 Jan 2023, 10:53:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 11 (0x0000000B) Unknown error code
Computer ID 1538037
Run time 16 hours 26 min
CPU time 16 hours 26 min
Validate state Invalid
Credit 8,830.08
Device peak FLOPS 4.76 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,756.67 MB
Peak swap size 4,978.08 MB
Peak disk usage 5,258.20 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process got signal 11</message>
<stderr_txt>
2189386/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_62.zip
Uploading the intermediate file: upload_file_62.zip
  07:03:03 STEP 1513 H=1513:00 +CPU= 22.976
Uploading trickle at timestep: 5443200
  07:03:29 STEP 1514 H=1514:00 +CPU= 25.145
  07:04:07 STEP 1515 H=1515:00 +CPU= 37.478
  07:04:33 STEP 1516 H=1516:00 +CPU= 24.293
  07:04:57 STEP 1517 H=1517:00 +CPU= 23.998
  07:05:35 STEP 1518 H=1518:00 +CPU= 37.748
  07:05:58 STEP 1519 H=1519:00 +CPU= 23.438
  07:06:23 STEP 1520 H=1520:00 +CPU= 24.248
  07:07:00 STEP 1521 H=1521:00 +CPU= 37.395
  07:07:24 STEP 1522 H=1522:00 +CPU= 23.668
  07:07:47 STEP 1523 H=1523:00 +CPU= 23.560
  07:08:28 STEP 1524 H=1524:00 +CPU= 40.246
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001524
  07:08:50 STEP 1525 H=1525:00 +CPU= 22.911
  07:09:15 STEP 1526 H=1526:00 +CPU= 24.229
  07:09:51 STEP 1527 H=1527:00 +CPU= 35.899
  07:10:14 STEP 1528 H=1528:00 +CPU= 23.491
  07:10:38 STEP 1529 H=1529:00 +CPU= 23.284
  07:11:16 STEP 1530 H=1530:00 +CPU= 38.650
  07:11:39 STEP 1531 H=1531:00 +CPU= 23.069
  07:12:04 STEP 1532 H=1532:00 +CPU= 24.980
  07:12:41 STEP 1533 H=1533:00 +CPU= 36.212
  07:13:04 STEP 1534 H=1534:00 +CPU= 23.353
  07:13:28 STEP 1535 H=1535:00 +CPU= 23.629
  07:14:11 STEP 1536 H=1536:00 +CPU= 43.197
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_63.zip
  07:14:34 STEP 1537 H=1537:00 +CPU= 22.581
Uploading the intermediate file: upload_file_63.zip
  07:14:57 STEP 1538 H=1538:00 +CPU= 23.462
  07:15:35 STEP 1539 H=1539:00 +CPU= 37.224
  07:15:58 STEP 1540 H=1540:00 +CPU= 23.411
  07:16:21 STEP 1541 H=1541:00 +CPU= 23.501
  07:17:00 STEP 1542 H=1542:00 +CPU= 38.184
  07:17:23 STEP 1543 H=1543:00 +CPU= 23.016
  07:17:47 STEP 1544 H=1544:00 +CPU= 24.109
  07:18:23 STEP 1545 H=1545:00 +CPU= 36.158
  07:18:46 STEP 1546 H=1546:00 +CPU= 23.503
  07:19:10 STEP 1547 H=1547:00 +CPU= 23.236
  07:19:50 STEP 1548 H=1548:00 +CPU= 40.355
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001548
  07:20:14 STEP 1549 H=1549:00 +CPU= 23.673
  07:20:37 STEP 1550 H=1550:00 +CPU= 23.509
  07:21:14 STEP 1551 H=1551:00 +CPU= 36.727
  07:21:38 STEP 1552 H=1552:00 +CPU= 23.683
  07:22:02 STEP 1553 H=1553:00 +CPU= 23.288
  07:22:39 STEP 1554 H=1554:00 +CPU= 37.209
  07:23:02 STEP 1555 H=1555:00 +CPU= 23.031
  07:23:27 STEP 1556 H=1556:00 +CPU= 24.560
  07:24:04 STEP 1557 H=1557:00 +CPU= 37.192
  07:24:30 STEP 1558 H=1558:00 +CPU= 24.851
  07:24:54 STEP 1559 H=1559:00 +CPU= 24.001
  07:25:37 STEP 1560 H=1560:00 +CPU= 43.159
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_64.zip
  07:26:00 STEP 1561 H=1561:00 +CPU= 23.074
Uploading the intermediate file: upload_file_64.zip
  07:26:24 STEP 1562 H=1562:00 +CPU= 23.482
  07:27:01 STEP 1563 H=1563:00 +CPU= 36.517
  07:27:25 STEP 1564 H=1564:00 +CPU= 23.410
  07:27:48 STEP 1565 H=1565:00 +CPU= 22.839
  07:28:25 STEP 1566 H=1566:00 +CPU= 37.092
  07:28:48 STEP 1567 H=1567:00 +CPU= 23.408
  07:29:14 STEP 1568 H=1568:00 +CPU= 25.893
  07:29:51 STEP 1569 H=1569:00 +CPU= 36.887
  07:30:17 STEP 1570 H=1570:00 +CPU= 25.419
  07:30:40 STEP 1571 H=1571:00 +CPU= 22.989
  07:31:20 STEP 1572 H=1572:00 +CPU= 40.412
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001572
  07:31:43 STEP 1573 H=1573:00 +CPU= 23.298
  07:32:07 STEP 1574 H=1574:00 +CPU= 23.725
  07:32:44 STEP 1575 H=1575:00 +CPU= 36.352
  07:33:08 STEP 1576 H=1576:00 +CPU= 23.801
  07:33:32 STEP 1577 H=1577:00 +CPU= 23.604
  07:34:09 STEP 1578 H=1578:00 +CPU= 37.441
  07:34:32 STEP 1579 H=1579:00 +CPU= 22.922
  07:34:57 STEP 1580 H=1580:00 +CPU= 25.026
  07:35:33 STEP 1581 H=1581:00 +CPU= 36.406
  07:35:58 STEP 1582 H=1582:00 +CPU= 24.306
  07:36:25 STEP 1583 H=1583:00 +CPU= 23.838
  07:37:09 STEP 1584 H=1584:00 +CPU= 42.901
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_65.zip
Uploading the intermediate file: upload_file_65.zip
  07:37:31 STEP 1585 H=1585:00 +CPU= 22.667
  07:37:55 STEP 1586 H=1586:00 +CPU= 23.951
  07:38:33 STEP 1587 H=1587:00 +CPU= 36.420
  07:38:56 STEP 1588 H=1588:00 +CPU= 23.136
  07:39:19 STEP 1589 H=1589:00 +CPU= 22.909
  07:39:57 STEP 1590 H=1590:00 +CPU= 37.649
  07:40:20 STEP 1591 H=1591:00 +CPU= 22.827
  07:40:44 STEP 1592 H=1592:00 +CPU= 23.969
  07:41:20 STEP 1593 H=1593:00 +CPU= 36.498
  07:41:44 STEP 1594 H=1594:00 +CPU= 23.841
  07:42:07 STEP 1595 H=1595:00 +CPU= 23.148
  07:42:50 STEP 1596 H=1596:00 +CPU= 42.566
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001596
  07:43:13 STEP 1597 H=1597:00 +CPU= 23.204
  07:43:37 STEP 1598 H=1598:00 +CPU= 24.117
  07:44:14 STEP 1599 H=1599:00 +CPU= 36.535
  07:44:40 STEP 1600 H=1600:00 +CPU= 25.386
  07:45:04 STEP 1601 H=1601:00 +CPU= 23.405
  07:45:42 STEP 1602 H=1602:00 +CPU= 37.976
  07:46:05 STEP 1603 H=1603:00 +CPU= 23.417
  07:46:29 STEP 1604 H=1604:00 +CPU= 24.104
  07:47:05 STEP 1605 H=1605:00 +CPU= 36.322
  07:47:33 STEP 1606 H=1606:00 +CPU= 25.608
  07:47:57 STEP 1607 H=1607:00 +CPU= 24.093
  07:48:44 STEP 1608 H=1608:00 +CPU= 43.008
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_66.zip
Uploading the intermediate file: upload_file_66.zip
  07:49:08 STEP 1609 H=1609:00 +CPU= 23.843
  07:49:31 STEP 1610 H=1610:00 +CPU= 23.539
  07:50:08 STEP 1611 H=1611:00 +CPU= 36.039
  07:50:31 STEP 1612 H=1612:00 +CPU= 23.437
  07:50:54 STEP 1613 H=1613:00 +CPU= 22.729
  07:51:32 STEP 1614 H=1614:00 +CPU= 37.482
  07:51:55 STEP 1615 H=1615:00 +CPU= 22.910
  07:52:18 STEP 1616 H=1616:00 +CPU= 23.536
  07:52:55 STEP 1617 H=1617:00 +CPU= 36.822
  07:53:19 STEP 1618 H=1618:00 +CPU= 23.575
  07:53:43 STEP 1619 H=1619:00 +CPU= 24.481
  07:54:23 STEP 1620 H=1620:00 +CPU= 39.805
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001620
  07:54:46 STEP 1621 H=1621:00 +CPU= 22.906
  07:55:11 STEP 1622 H=1622:00 +CPU= 24.932
  07:55:49 STEP 1623 H=1623:00 +CPU= 37.991
  07:56:13 STEP 1624 H=1624:00 +CPU= 23.686
  07:56:37 STEP 1625 H=1625:00 +CPU= 22.619
  07:57:14 STEP 1626 H=1626:00 +CPU= 37.112
  07:57:37 STEP 1627 H=1627:00 +CPU= 22.656
  07:58:00 STEP 1628 H=1628:00 +CPU= 23.306
  07:58:37 STEP 1629 H=1629:00 +CPU= 36.321
  07:59:00 STEP 1630 H=1630:00 +CPU= 23.182
  07:59:23 STEP 1631 H=1631:00 +CPU= 23.464
  08:00:08 STEP 1632 H=1632:00 +CPU= 44.798
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_67.zip
  08:00:33 STEP 1633 H=1633:00 +CPU= 24.169
Uploading the intermediate file: upload_file_67.zip
  08:00:57 STEP 1634 H=1634:00 +CPU= 23.882
  08:01:36 STEP 1635 H=1635:00 +CPU= 39.012
  08:02:01 STEP 1636 H=1636:00 +CPU= 23.796
  08:02:23 STEP 1637 H=1637:00 +CPU= 22.860
  08:03:01 STEP 1638 H=1638:00 +CPU= 37.579
  08:03:25 STEP 1639 H=1639:00 +CPU= 24.136
  08:03:50 STEP 1640 H=1640:00 +CPU= 23.700
  08:04:26 STEP 1641 H=1641:00 +CPU= 36.056
  08:04:49 STEP 1642 H=1642:00 +CPU= 23.584
  08:05:13 STEP 1643 H=1643:00 +CPU= 23.679
  08:06:05 STEP 1644 H=1644:00 +CPU= 43.801
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001644
  08:06:28 STEP 1645 H=1645:00 +CPU= 23.338
  08:06:52 STEP 1646 H=1646:00 +CPU= 24.267
  08:07:30 STEP 1647 H=1647:00 +CPU= 37.315
  08:07:54 STEP 1648 H=1648:00 +CPU= 23.867
  08:08:16 STEP 1649 H=1649:00 +CPU= 22.466
  08:08:54 STEP 1650 H=1650:00 +CPU= 37.470
  08:09:24 STEP 1651 H=1651:00 +CPU= 22.885
  08:09:53 STEP 1652 H=1652:00 +CPU= 23.463
  08:10:31 STEP 1653 H=1653:00 +CPU= 35.597
  08:10:55 STEP 1654 H=1654:00 +CPU= 23.686
  08:11:19 STEP 1655 H=1655:00 +CPU= 24.133
  08:12:03 STEP 1656 H=1656:00 +CPU= 43.906
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_68.zip
Uploading the intermediate file: upload_file_68.zip
  08:12:27 STEP 1657 H=1657:00 +CPU= 23.201
  08:12:51 STEP 1658 H=1658:00 +CPU= 23.538
  08:13:27 STEP 1659 H=1659:00 +CPU= 35.436
  08:13:51 STEP 1660 H=1660:00 +CPU= 23.844
  08:14:15 STEP 1661 H=1661:00 +CPU= 23.380
  08:14:52 STEP 1662 H=1662:00 +CPU= 36.829
  08:15:16 STEP 1663 H=1663:00 +CPU= 24.233
  08:15:41 STEP 1664 H=1664:00 +CPU= 23.770
  08:16:18 STEP 1665 H=1665:00 +CPU= 35.764
  08:16:42 STEP 1666 H=1666:00 +CPU= 23.796
  08:17:05 STEP 1667 H=1667:00 +CPU= 22.884
  08:17:46 STEP 1668 H=1668:00 +CPU= 40.085
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001668
  08:18:09 STEP 1669 H=1669:00 +CPU= 22.604
  08:18:32 STEP 1670 H=1670:00 +CPU= 23.289
  08:19:08 STEP 1671 H=1671:00 +CPU= 36.006
  08:19:31 STEP 1672 H=1672:00 +CPU= 23.106
  08:19:54 STEP 1673 H=1673:00 +CPU= 22.525
  08:20:32 STEP 1674 H=1674:00 +CPU= 36.636
  08:20:54 STEP 1675 H=1675:00 +CPU= 22.761
  08:21:18 STEP 1676 H=1676:00 +CPU= 23.162
  08:21:55 STEP 1677 H=1677:00 +CPU= 36.314
  08:22:18 STEP 1678 H=1678:00 +CPU= 23.504
  08:22:42 STEP 1679 H=1679:00 +CPU= 23.477
  08:23:27 STEP 1680 H=1680:00 +CPU= 43.260
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_69.zip
Uploading the intermediate file: upload_file_69.zip
  08:23:51 STEP 1681 H=1681:00 +CPU= 23.705
  08:24:14 STEP 1682 H=1682:00 +CPU= 23.078
  08:24:50 STEP 1683 H=1683:00 +CPU= 36.037
  08:25:15 STEP 1684 H=1684:00 +CPU= 23.290
  08:25:38 STEP 1685 H=1685:00 +CPU= 22.905
  08:26:16 STEP 1686 H=1686:00 +CPU= 37.296
  08:26:39 STEP 1687 H=1687:00 +CPU= 22.896
  08:27:03 STEP 1688 H=1688:00 +CPU= 23.597
  08:27:39 STEP 1689 H=1689:00 +CPU= 36.253
  08:28:02 STEP 1690 H=1690:00 +CPU= 23.568
  08:28:25 STEP 1691 H=1691:00 +CPU= 22.784
  08:29:06 STEP 1692 H=1692:00 +CPU= 40.128
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001692
  08:29:30 STEP 1693 H=1693:00 +CPU= 23.595
  08:29:53 STEP 1694 H=1694:00 +CPU= 23.175
  08:30:29 STEP 1695 H=1695:00 +CPU= 36.126
  08:30:53 STEP 1696 H=1696:00 +CPU= 23.726
  08:31:16 STEP 1697 H=1697:00 +CPU= 22.682
  08:31:54 STEP 1698 H=1698:00 +CPU= 37.356
  08:32:18 STEP 1699 H=1699:00 +CPU= 23.157
  08:32:42 STEP 1700 H=1700:00 +CPU= 23.416
  08:33:18 STEP 1701 H=1701:00 +CPU= 36.490
  08:33:42 STEP 1702 H=1702:00 +CPU= 23.928
  08:34:07 STEP 1703 H=1703:00 +CPU= 24.814
  08:34:52 STEP 1704 H=1704:00 +CPU= 43.994
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_70.zip
  08:35:14 STEP 1705 H=1705:00 +CPU= 22.846
Uploading the intermediate file: upload_file_70.zip
  08:35:38 STEP 1706 H=1706:00 +CPU= 23.928
  08:36:16 STEP 1707 H=1707:00 +CPU= 36.727
  08:36:40 STEP 1708 H=1708:00 +CPU= 23.969
  08:37:03 STEP 1709 H=1709:00 +CPU= 23.307
  08:37:42 STEP 1710 H=1710:00 +CPU= 38.711
  08:38:05 STEP 1711 H=1711:00 +CPU= 23.075
  08:38:29 STEP 1712 H=1712:00 +CPU= 23.281
  08:39:06 STEP 1713 H=1713:00 +CPU= 36.642
  08:39:30 STEP 1714 H=1714:00 +CPU= 23.706
  08:39:53 STEP 1715 H=1715:00 +CPU= 23.090
  08:40:33 STEP 1716 H=1716:00 +CPU= 40.585
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001716
  08:40:56 STEP 1717 H=1717:00 +CPU= 22.875
  08:41:20 STEP 1718 H=1718:00 +CPU= 23.770
  08:41:57 STEP 1719 H=1719:00 +CPU= 36.953
  08:42:21 STEP 1720 H=1720:00 +CPU= 24.116
  08:42:47 STEP 1721 H=1721:00 +CPU= 24.475
  08:43:25 STEP 1722 H=1722:00 +CPU= 38.068
  08:43:48 STEP 1723 H=1723:00 +CPU= 22.968
  08:44:12 STEP 1724 H=1724:00 +CPU= 23.558
  08:44:49 STEP 1725 H=1725:00 +CPU= 37.550
  08:45:14 STEP 1726 H=1726:00 +CPU= 25.139
  08:45:41 STEP 1727 H=1727:00 +CPU= 23.975
  08:46:28 STEP 1728 H=1728:00 +CPU= 45.063
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_71.zip
Uploading the intermediate file: upload_file_71.zip
  08:46:51 STEP 1729 H=1729:00 +CPU= 22.942
  08:47:14 STEP 1730 H=1730:00 +CPU= 23.108
  08:47:50 STEP 1731 H=1731:00 +CPU= 36.642
  08:48:14 STEP 1732 H=1732:00 +CPU= 23.758
  08:48:37 STEP 1733 H=1733:00 +CPU= 22.827
  08:49:15 STEP 1734 H=1734:00 +CPU= 37.083
  08:49:38 STEP 1735 H=1735:00 +CPU= 23.325
  08:50:02 STEP 1736 H=1736:00 +CPU= 23.297
  08:50:40 STEP 1737 H=1737:00 +CPU= 36.068
  08:51:04 STEP 1738 H=1738:00 +CPU= 24.248
  08:51:27 STEP 1739 H=1739:00 +CPU= 22.709
  08:52:07 STEP 1740 H=1740:00 +CPU= 40.235
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001740
  08:52:31 STEP 1741 H=1741:00 +CPU= 23.822
  08:52:55 STEP 1742 H=1742:00 +CPU= 23.936
  08:53:32 STEP 1743 H=1743:00 +CPU= 35.987
  08:53:56 STEP 1744 H=1744:00 +CPU= 24.049
  08:54:19 STEP 1745 H=1745:00 +CPU= 22.660
  08:54:57 STEP 1746 H=1746:00 +CPU= 37.516
  08:55:20 STEP 1747 H=1747:00 +CPU= 23.074
  08:55:45 STEP 1748 H=1748:00 +CPU= 24.135
  08:56:21 STEP 1749 H=1749:00 +CPU= 36.759
  08:56:46 STEP 1750 H=1750:00 +CPU= 24.430
  08:57:10 STEP 1751 H=1751:00 +CPU= 23.925
  08:57:53 STEP 1752 H=1752:00 +CPU= 42.840
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_72.zip
Uploading the intermediate file: upload_file_72.zip
  08:58:15 STEP 1753 H=1753:00 +CPU= 22.853
Uploading trickle at timestep: 6307200
  08:58:39 STEP 1754 H=1754:00 +CPU= 23.757
  08:59:15 STEP 1755 H=1755:00 +CPU= 36.038
  08:59:40 STEP 1756 H=1756:00 +CPU= 23.996
  09:00:03 STEP 1757 H=1757:00 +CPU= 22.751
  09:00:40 STEP 1758 H=1758:00 +CPU= 37.037
  09:01:03 STEP 1759 H=1759:00 +CPU= 23.091
  09:01:27 STEP 1760 H=1760:00 +CPU= 23.855
  09:02:03 STEP 1761 H=1761:00 +CPU= 36.506
  09:02:30 STEP 1762 H=1762:00 +CPU= 26.643
  09:02:54 STEP 1763 H=1763:00 +CPU= 23.194
  09:03:34 STEP 1764 H=1764:00 +CPU= 40.415
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001764
  09:03:58 STEP 1765 H=1765:00 +CPU= 23.115
  09:04:21 STEP 1766 H=1766:00 +CPU= 23.472
  09:04:58 STEP 1767 H=1767:00 +CPU= 36.988
  09:05:22 STEP 1768 H=1768:00 +CPU= 23.637
  09:05:44 STEP 1769 H=1769:00 +CPU= 22.693
  09:06:23 STEP 1770 H=1770:00 +CPU= 38.844
  09:06:47 STEP 1771 H=1771:00 +CPU= 23.337
  09:07:11 STEP 1772 H=1772:00 +CPU= 23.667
  09:07:47 STEP 1773 H=1773:00 +CPU= 36.516
  09:08:11 STEP 1774 H=1774:00 +CPU= 23.646
  09:08:34 STEP 1775 H=1775:00 +CPU= 23.174
  09:09:21 STEP 1776 H=1776:00 +CPU= 45.005
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_73.zip
Uploading the intermediate file: upload_file_73.zip
  09:09:46 STEP 1777 H=1777:00 +CPU= 25.659
  09:10:11 STEP 1778 H=1778:00 +CPU= 24.474
  09:10:51 STEP 1779 H=1779:00 +CPU= 40.331
  09:11:16 STEP 1780 H=1780:00 +CPU= 24.015
  09:11:39 STEP 1781 H=1781:00 +CPU= 23.000
  09:12:15 STEP 1782 H=1782:00 +CPU= 36.659
  09:12:39 STEP 1783 H=1783:00 +CPU= 23.924
  09:13:05 STEP 1784 H=1784:00 +CPU= 24.591
  09:13:44 STEP 1785 H=1785:00 +CPU= 39.140
  09:14:08 STEP 1786 H=1786:00 +CPU= 24.516
  09:14:31 STEP 1787 H=1787:00 +CPU= 22.640
  09:15:14 STEP 1788 H=1788:00 +CPU= 42.794
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001788
  09:15:38 STEP 1789 H=1789:00 +CPU= 23.712
  09:16:03 STEP 1790 H=1790:00 +CPU= 23.903
  09:16:40 STEP 1791 H=1791:00 +CPU= 36.802
  09:17:05 STEP 1792 H=1792:00 +CPU= 24.734
  09:17:28 STEP 1793 H=1793:00 +CPU= 22.804
  09:18:05 STEP 1794 H=1794:00 +CPU= 36.661
  09:18:28 STEP 1795 H=1795:00 +CPU= 22.714
  09:18:52 STEP 1796 H=1796:00 +CPU= 23.524
  09:19:28 STEP 1797 H=1797:00 +CPU= 36.329
  09:19:53 STEP 1798 H=1798:00 +CPU= 24.154
  09:20:17 STEP 1799 H=1799:00 +CPU= 24.815
  09:21:02 STEP 1800 H=1800:00 +CPU= 44.455
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_74.zip
Uploading the intermediate file: upload_file_74.zip
  09:21:25 STEP 1801 H=1801:00 +CPU= 22.801
  09:21:50 STEP 1802 H=1802:00 +CPU= 25.152
  09:22:27 STEP 1803 H=1803:00 +CPU= 37.332
  09:22:52 STEP 1804 H=1804:00 +CPU= 25.079
  09:23:16 STEP 1805 H=1805:00 +CPU= 23.757
  09:23:53 STEP 1806 H=1806:00 +CPU= 36.987
  09:24:16 STEP 1807 H=1807:00 +CPU= 23.049
  09:24:40 STEP 1808 H=1808:00 +CPU= 23.687
  09:25:17 STEP 1809 H=1809:00 +CPU= 37.011
  09:25:43 STEP 1810 H=1810:00 +CPU= 25.891
  09:26:06 STEP 1811 H=1811:00 +CPU= 23.369
  09:26:47 STEP 1812 H=1812:00 +CPU= 41.071
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001812
  09:27:10 STEP 1813 H=1813:00 +CPU= 22.972
  09:27:35 STEP 1814 H=1814:00 +CPU= 24.383
  09:28:15 STEP 1815 H=1815:00 +CPU= 39.934
  09:28:42 STEP 1816 H=1816:00 +CPU= 25.837
  09:29:06 STEP 1817 H=1817:00 +CPU= 23.870
  09:29:43 STEP 1818 H=1818:00 +CPU= 37.049
  09:30:06 STEP 1819 H=1819:00 +CPU= 22.811
  09:30:31 STEP 1820 H=1820:00 +CPU= 24.508
  09:31:09 STEP 1821 H=1821:00 +CPU= 37.510
  09:31:34 STEP 1822 H=1822:00 +CPU= 25.724
  09:32:02 STEP 1823 H=1823:00 +CPU= 27.760
  09:32:49 STEP 1824 H=1824:00 +CPU= 46.767
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_75.zip
Uploading the intermediate file: upload_file_75.zip
  09:33:12 STEP 1825 H=1825:00 +CPU= 23.153
  09:33:36 STEP 1826 H=1826:00 +CPU= 23.569
  09:34:16 STEP 1827 H=1827:00 +CPU= 36.275
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0742
(argv4) batchid: 972
(argv5) wuid: 12189386
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_abf6335097398061f2414da3967a64e8 to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0742_2003050100_123_972_12189386.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0742_2003050100_123_972_12189386.zip
ic_ancil_file: ic_ancil_12189386
ifsdata_file: ifsdata_12189386
climate_data_file: clim_data_12189386
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_94e371871c8c7ecc4c71651a3bb73ecc to: /var/lib/boinc-client/slots/13/ic_ancil_12189386.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12189386.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4bab10b542cf40ea0ffa7679f3f497b2 to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12189386.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12189386.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12189386.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12189386.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12189386.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12189386.xml
-- Model is restarting --
Adjusting last_iter, 1827, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12189386.xml
last_cpu_time: 55116
upload_file_number: 76
last_iter: 1823
last_upload: 6566400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428
The child process has been launched with process id: 156741
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:156741:156741] [20230118:093630:1674002190:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:37:53 STEP 1824 H=1824:00 +CPU= 43.232
  09:38:17 STEP 1825 H=1825:00 +CPU= 22.810
  09:38:40 STEP 1826 H=1826:00 +CPU= 22.997
  09:39:15 STEP 1827 H=1827:00 +CPU= 34.823
  09:39:40 STEP 1828 H=1828:00 +CPU= 22.867
  09:40:03 STEP 1829 H=1829:00 +CPU= 22.387
  09:40:52 STEP 1830 H=1830:00 +CPU= 35.460
  09:41:14 STEP 1831 H=1831:00 +CPU= 22.209
  09:41:36 STEP 1832 H=1832:00 +CPU= 22.551
  09:42:23 STEP 1833 H=1833:00 +CPU= 35.614
  09:43:07 STEP 1834 H=1834:00 +CPU= 23.339
  09:43:33 STEP 1835 H=1835:00 +CPU= 23.482
  09:44:23 STEP 1836 H=1836:00 +CPU= 40.014
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001836
  09:44:45 STEP 1837 H=1837:00 +CPU= 22.449
  09:45:08 STEP 1838 H=1838:00 +CPU= 23.001
  09:45:43 STEP 1839 H=1839:00 +CPU= 34.990
  09:46:06 STEP 1840 H=1840:00 +CPU= 22.965
  09:46:29 STEP 1841 H=1841:00 +CPU= 22.144
  09:47:05 STEP 1842 H=1842:00 +CPU= 35.486
  09:47:26 STEP 1843 H=1843:00 +CPU= 21.775
  09:47:50 STEP 1844 H=1844:00 +CPU= 22.570
  09:48:26 STEP 1845 H=1845:00 +CPU= 35.188
  09:48:49 STEP 1846 H=1846:00 +CPU= 22.606
  09:49:12 STEP 1847 H=1847:00 +CPU= 22.730
  09:50:20 STEP 1848 H=1848:00 +CPU= 41.826
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_76.zip
  09:50:43 STEP 1849 H=1849:00 +CPU= 22.385
Uploading the intermediate file: upload_file_76.zip
  09:51:07 STEP 1850 H=1850:00 +CPU= 23.793
  09:51:44 STEP 1851 H=1851:00 +CPU= 35.221
  09:52:06 STEP 1852 H=1852:00 +CPU= 22.087
  09:52:28 STEP 1853 H=1853:00 +CPU= 21.471
  09:53:06 STEP 1854 H=1854:00 +CPU= 37.456
  09:53:30 STEP 1855 H=1855:00 +CPU= 23.113
  09:53:53 STEP 1856 H=1856:00 +CPU= 22.764
  09:54:31 STEP 1857 H=1857:00 +CPU= 37.700
  09:54:54 STEP 1858 H=1858:00 +CPU= 23.316
  09:55:17 STEP 1859 H=1859:00 +CPU= 22.658
  09:56:02 STEP 1860 H=1860:00 +CPU= 41.853
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001860
  09:56:25 STEP 1861 H=1861:00 +CPU= 23.346
  09:56:49 STEP 1862 H=1862:00 +CPU= 23.639
  09:57:26 STEP 1863 H=1863:00 +CPU= 36.697
  09:57:49 STEP 1864 H=1864:00 +CPU= 23.474
  09:58:12 STEP 1865 H=1865:00 +CPU= 22.697
  09:58:49 STEP 1866 H=1866:00 +CPU= 36.630
  09:59:12 STEP 1867 H=1867:00 +CPU= 22.292
  09:59:36 STEP 1868 H=1868:00 +CPU= 23.351
  10:00:12 STEP 1869 H=1869:00 +CPU= 35.846
  10:00:36 STEP 1870 H=1870:00 +CPU= 23.157
  10:01:05 STEP 1871 H=1871:00 +CPU= 23.703
  10:01:59 STEP 1872 H=1872:00 +CPU= 44.341
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_77.zip
Uploading the intermediate file: upload_file_77.zip
  10:02:23 STEP 1873 H=1873:00 +CPU= 23.957
  10:02:46 STEP 1874 H=1874:00 +CPU= 23.417
  10:03:22 STEP 1875 H=1875:00 +CPU= 35.868
  10:03:46 STEP 1876 H=1876:00 +CPU= 23.522
  10:04:09 STEP 1877 H=1877:00 +CPU= 23.202
  10:04:46 STEP 1878 H=1878:00 +CPU= 37.020
  10:05:10 STEP 1879 H=1879:00 +CPU= 23.898
  10:05:34 STEP 1880 H=1880:00 +CPU= 23.918
  10:06:12 STEP 1881 H=1881:00 +CPU= 37.123
  10:06:37 STEP 1882 H=1882:00 +CPU= 24.132
  10:07:01 STEP 1883 H=1883:00 +CPU= 23.002
  10:07:42 STEP 1884 H=1884:00 +CPU= 40.233
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001884
  10:08:05 STEP 1885 H=1885:00 +CPU= 22.914
  10:08:29 STEP 1886 H=1886:00 +CPU= 23.592
  10:09:05 STEP 1887 H=1887:00 +CPU= 36.535
  10:09:29 STEP 1888 H=1888:00 +CPU= 23.644
  10:09:52 STEP 1889 H=1889:00 +CPU= 22.630
  10:10:30 STEP 1890 H=1890:00 +CPU= 37.107
  10:10:52 STEP 1891 H=1891:00 +CPU= 22.646
  10:11:17 STEP 1892 H=1892:00 +CPU= 24.364
  10:11:54 STEP 1893 H=1893:00 +CPU= 36.361
  10:12:17 STEP 1894 H=1894:00 +CPU= 22.417
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2003050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0742
(argv4) batchid: 972
(argv5) wuid: 12189386
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_abf6335097398061f2414da3967a64e8 to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0742_2003050100_123_972_12189386.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0742_2003050100_123_972_12189386.zip
ic_ancil_file: ic_ancil_12189386
ifsdata_file: ifsdata_12189386
climate_data_file: clim_data_12189386
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_94e371871c8c7ecc4c71651a3bb73ecc to: /var/lib/boinc-client/slots/13/ic_ancil_12189386.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12189386.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4bab10b542cf40ea0ffa7679f3f497b2 to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12189386.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12189386.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12189386.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12189386.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12189386.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12189386.xml
-- Model is restarting --
Adjusting last_iter, 1895, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12189386.xml
last_cpu_time: 57120
upload_file_number: 78
last_iter: 1871
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428
The child process has been launched with process id: 3134
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:3134:3134] [20230118:111941:1674008381:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:22:58 STEP 1872 H=1872:00 +CPU= 41.974
  11:23:21 STEP 1873 H=1873:00 +CPU= 22.884
  11:23:44 STEP 1874 H=1874:00 +CPU= 22.619
  11:24:19 STEP 1875 H=1875:00 +CPU= 35.014
  11:24:51 STEP 1876 H=1876:00 +CPU= 21.748
  11:25:13 STEP 1877 H=1877:00 +CPU= 21.171
  11:26:11 STEP 1878 H=1878:00 +CPU= 35.129
  11:26:38 STEP 1879 H=1879:00 +CPU= 22.119
  11:27:00 STEP 1880 H=1880:00 +CPU= 22.213
  11:27:47 STEP 1881 H=1881:00 +CPU= 34.872
  11:28:11 STEP 1882 H=1882:00 +CPU= 23.230
  11:28:33 STEP 1883 H=1883:00 +CPU= 22.566
  11:29:13 STEP 1884 H=1884:00 +CPU= 40.253
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001884
  11:29:36 STEP 1885 H=1885:00 +CPU= 22.411
  11:30:00 STEP 1886 H=1886:00 +CPU= 24.045
  11:30:36 STEP 1887 H=1887:00 +CPU= 35.908
  11:31:00 STEP 1888 H=1888:00 +CPU= 24.268
  11:31:23 STEP 1889 H=1889:00 +CPU= 23.020
  11:32:01 STEP 1890 H=1890:00 +CPU= 37.926
  11:32:24 STEP 1891 H=1891:00 +CPU= 22.728
  11:32:48 STEP 1892 H=1892:00 +CPU= 23.784
  11:33:25 STEP 1893 H=1893:00 +CPU= 36.908
  11:33:48 STEP 1894 H=1894:00 +CPU= 23.014
  11:34:24 STEP 1895 H=1895:00 +CPU= 24.190
  11:37:19 STEP 1896 H=1896:00 +CPU= 43.881
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_78.zip
  11:37:41 STEP 1897 H=1897:00 +CPU= 22.369
  11:38:04 STEP 1898 H=1898:00 +CPU= 22.951
Uploading the intermediate file: upload_file_78.zip
  11:38:43 STEP 1899 H=1899:00 +CPU= 38.125
  11:39:06 STEP 1900 H=1900:00 +CPU= 23.467
  11:39:28 STEP 1901 H=1901:00 +CPU= 21.378
  11:40:04 STEP 1902 H=1902:00 +CPU= 35.133
  11:40:26 STEP 1903 H=1903:00 +CPU= 21.657
  11:40:48 STEP 1904 H=1904:00 +CPU= 22.412
  11:41:23 STEP 1905 H=1905:00 +CPU= 34.716
  11:41:46 STEP 1906 H=1906:00 +CPU= 22.166
  11:42:07 STEP 1907 H=1907:00 +CPU= 21.688
  11:42:48 STEP 1908 H=1908:00 +CPU= 39.953
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001908
  11:43:12 STEP 1909 H=1909:00 +CPU= 23.393
  11:43:35 STEP 1910 H=1910:00 +CPU= 23.161
  11:44:11 STEP 1911 H=1911:00 +CPU= 35.886
  11:44:34 STEP 1912 H=1912:00 +CPU= 23.444
  11:44:57 STEP 1913 H=1913:00 +CPU= 22.588
  11:45:34 STEP 1914 H=1914:00 +CPU= 36.253
  11:45:57 STEP 1915 H=1915:00 +CPU= 22.445
  11:46:20 STEP 1916 H=1916:00 +CPU= 23.317
  11:46:56 STEP 1917 H=1917:00 +CPU= 35.893
  11:47:19 STEP 1918 H=1918:00 +CPU= 23.132
  11:47:43 STEP 1919 H=1919:00 +CPU= 23.974
  11:48:25 STEP 1920 H=1920:00 +CPU= 42.032
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_79.zip
  11:48:48 STEP 1921 H=1921:00 +CPU= 22.271
  11:49:11 STEP 1922 H=1922:00 +CPU= 23.181
Uploading the intermediate file: upload_file_79.zip
  11:49:46 STEP 1923 H=1923:00 +CPU= 35.620
  11:50:09 STEP 1924 H=1924:00 +CPU= 22.874
  11:50:32 STEP 1925 H=1925:00 +CPU= 22.483
  11:51:09 STEP 1926 H=1926:00 +CPU= 36.750
  11:51:32 STEP 1927 H=1927:00 +CPU= 22.444
  11:51:55 STEP 1928 H=1928:00 +CPU= 23.407
  11:52:31 STEP 1929 H=1929:00 +CPU= 35.960
  11:52:56 STEP 1930 H=1930:00 +CPU= 23.895
  11:53:19 STEP 1931 H=1931:00 +CPU= 22.857
  11:53:59 STEP 1932 H=1932:00 +CPU= 39.959
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001932
  11:54:22 STEP 1933 H=1933:00 +CPU= 22.523
  11:54:45 STEP 1934 H=1934:00 +CPU= 23.032
  11:55:21 STEP 1935 H=1935:00 +CPU= 35.830
  11:55:45 STEP 1936 H=1936:00 +CPU= 23.461
  11:56:08 STEP 1937 H=1937:00 +CPU= 22.637
  11:56:44 STEP 1938 H=1938:00 +CPU= 36.471
  11:57:07 STEP 1939 H=1939:00 +CPU= 22.660
  11:57:30 STEP 1940 H=1940:00 +CPU= 23.258
  11:58:06 STEP 1941 H=1941:00 +CPU= 35.733
  11:58:29 STEP 1942 H=1942:00 +CPU= 23.040
  11:58:52 STEP 1943 H=1943:00 +CPU= 23.221
  11:59:34 STEP 1944 H=1944:00 +CPU= 41.887
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189386/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0742_2003050100_123_972_12189386_0_r1644802428_80.zip
Uploading the intermediate file: upload_file_80.zip
  11:59:57 STEP 1945 H=1945:00 +CPU= 22.365

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:25:12 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 6,307,200 52,957 0.0084
09 May 2023 16:25:01 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 5,443,200 46,119 0.0085
09 May 2023 16:25:01 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 5,443,200 46,119 0.0085
09 May 2023 16:24:45 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 4,492,800 38,014 0.0085
09 May 2023 16:24:35 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 3,628,800 30,721 0.0085
09 May 2023 16:24:25 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 2,678,400 22,220 0.0083
09 May 2023 16:24:15 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 1,814,400 14,949 0.0082
09 May 2023 16:24:00 1538037 22274922 oifs_43r3_ps_0742_2003050100_123_972_12189386_0 864,000 6,889 0.0080


©2024 climateprediction.net