climateprediction.net home page
Task 22264812

Task 22264812

Name oifs_43r3_ps_0234_1994050100_123_963_12179878_0
Workunit 12179878
Created 22 Dec 2022, 5:12:43 UTC
Sent 5 Jan 2023, 1:55:34 UTC
Report deadline 4 Feb 2023, 1:55:34 UTC
Received 6 Jan 2023, 7:59:21 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1533065
Run time 22 hours 49 min 58 sec
CPU time 21 hours 51 min 57 sec
Validate state Invalid
Credit 10,886.40
Device peak FLOPS 5.35 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.29 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,323.22 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
cts/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_70.zip
Uploading the intermediate file: upload_file_70.zip
  03:39:11 STEP 1705 H=1705:00 +CPU= 28.823
  03:39:45 STEP 1706 H=1706:00 +CPU= 32.260
  03:40:39 STEP 1707 H=1707:00 +CPU= 50.541
  03:41:20 STEP 1708 H=1708:00 +CPU= 37.203
  03:41:58 STEP 1709 H=1709:00 +CPU= 36.813
  03:42:50 STEP 1710 H=1710:00 +CPU= 50.694
  03:43:26 STEP 1711 H=1711:00 +CPU= 32.710
  03:44:01 STEP 1712 H=1712:00 +CPU= 33.971
  03:44:52 STEP 1713 H=1713:00 +CPU= 48.891
  03:45:27 STEP 1714 H=1714:00 +CPU= 33.112
  03:46:01 STEP 1715 H=1715:00 +CPU= 32.137
  03:47:02 STEP 1716 H=1716:00 +CPU= 58.247
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001716
  03:47:40 STEP 1717 H=1717:00 +CPU= 36.152
  03:48:17 STEP 1718 H=1718:00 +CPU= 36.382
  03:49:15 STEP 1719 H=1719:00 +CPU= 56.338
  03:49:55 STEP 1720 H=1720:00 +CPU= 37.554
  03:50:38 STEP 1721 H=1721:00 +CPU= 41.316
  03:51:40 STEP 1722 H=1722:00 +CPU= 60.295
  03:52:18 STEP 1723 H=1723:00 +CPU= 36.603
  03:52:56 STEP 1724 H=1724:00 +CPU= 37.621
  03:53:52 STEP 1725 H=1725:00 +CPU= 53.297
  03:54:27 STEP 1726 H=1726:00 +CPU= 33.606
  03:55:11 STEP 1727 H=1727:00 +CPU= 41.181
  03:56:14 STEP 1728 H=1728:00 +CPU= 61.404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_71.zip
Uploading the intermediate file: upload_file_71.zip
  03:56:51 STEP 1729 H=1729:00 +CPU= 34.466
  03:57:28 STEP 1730 H=1730:00 +CPU= 36.283
  03:58:27 STEP 1731 H=1731:00 +CPU= 54.191
  03:59:02 STEP 1732 H=1732:00 +CPU= 32.907
  03:59:40 STEP 1733 H=1733:00 +CPU= 36.181
  04:00:38 STEP 1734 H=1734:00 +CPU= 54.674
  04:01:17 STEP 1735 H=1735:00 +CPU= 35.395
  04:01:56 STEP 1736 H=1736:00 +CPU= 38.001
  04:02:53 STEP 1737 H=1737:00 +CPU= 54.469
  04:03:29 STEP 1738 H=1738:00 +CPU= 34.074
  04:04:02 STEP 1739 H=1739:00 +CPU= 30.635
  04:04:59 STEP 1740 H=1740:00 +CPU= 53.980
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001740
  04:05:38 STEP 1741 H=1741:00 +CPU= 35.821
  04:06:15 STEP 1742 H=1742:00 +CPU= 36.176
  04:07:13 STEP 1743 H=1743:00 +CPU= 55.768
  04:07:51 STEP 1744 H=1744:00 +CPU= 36.384
  04:08:27 STEP 1745 H=1745:00 +CPU= 34.250
  04:09:23 STEP 1746 H=1746:00 +CPU= 54.163
  04:10:05 STEP 1747 H=1747:00 +CPU= 39.216
  04:10:47 STEP 1748 H=1748:00 +CPU= 41.483
  04:11:51 STEP 1749 H=1749:00 +CPU= 60.724
  04:12:33 STEP 1750 H=1750:00 +CPU= 40.183
  04:13:15 STEP 1751 H=1751:00 +CPU= 40.604
  04:14:23 STEP 1752 H=1752:00 +CPU= 65.859
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_72.zip
Uploading the intermediate file: upload_file_72.zip
  04:14:58 STEP 1753 H=1753:00 +CPU= 32.240
  04:15:35 STEP 1754 H=1754:00 +CPU= 35.018
  04:16:29 STEP 1755 H=1755:00 +CPU= 50.711
  04:17:04 STEP 1756 H=1756:00 +CPU= 33.221
  04:17:37 STEP 1757 H=1757:00 +CPU= 31.575
  04:18:29 STEP 1758 H=1758:00 +CPU= 49.548
  04:19:02 STEP 1759 H=1759:00 +CPU= 31.598
  04:19:34 STEP 1760 H=1760:00 +CPU= 30.806
  04:20:25 STEP 1761 H=1761:00 +CPU= 49.017
  04:21:02 STEP 1762 H=1762:00 +CPU= 35.556
  04:21:39 STEP 1763 H=1763:00 +CPU= 35.580
  04:22:44 STEP 1764 H=1764:00 +CPU= 59.591
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001764
  04:23:22 STEP 1765 H=1765:00 +CPU= 36.013
  04:23:59 STEP 1766 H=1766:00 +CPU= 35.372
  04:24:51 STEP 1767 H=1767:00 +CPU= 48.765
  04:25:26 STEP 1768 H=1768:00 +CPU= 33.217
  04:25:59 STEP 1769 H=1769:00 +CPU= 30.562
  04:26:49 STEP 1770 H=1770:00 +CPU= 49.251
  04:27:20 STEP 1771 H=1771:00 +CPU= 29.578
  04:27:52 STEP 1772 H=1772:00 +CPU= 29.935
  04:28:39 STEP 1773 H=1773:00 +CPU= 43.948
  04:29:10 STEP 1774 H=1774:00 +CPU= 29.858
  04:29:40 STEP 1775 H=1775:00 +CPU= 29.378
  04:30:39 STEP 1776 H=1776:00 +CPU= 56.555
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_73.zip
Uploading the intermediate file: upload_file_73.zip
  04:31:14 STEP 1777 H=1777:00 +CPU= 32.850
  04:31:49 STEP 1778 H=1778:00 +CPU= 33.790
  04:32:39 STEP 1779 H=1779:00 +CPU= 47.891
  04:33:09 STEP 1780 H=1780:00 +CPU= 29.269
  04:33:39 STEP 1781 H=1781:00 +CPU= 28.238
  04:34:26 STEP 1782 H=1782:00 +CPU= 45.328
  04:34:56 STEP 1783 H=1783:00 +CPU= 28.698
  04:35:26 STEP 1784 H=1784:00 +CPU= 29.407
  04:36:11 STEP 1785 H=1785:00 +CPU= 43.559
  04:36:41 STEP 1786 H=1786:00 +CPU= 29.576
  04:37:13 STEP 1787 H=1787:00 +CPU= 28.732
  04:38:03 STEP 1788 H=1788:00 +CPU= 48.193
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001788
  04:38:36 STEP 1789 H=1789:00 +CPU= 31.902
  04:39:10 STEP 1790 H=1790:00 +CPU= 33.303
  04:39:58 STEP 1791 H=1791:00 +CPU= 46.877
  04:40:34 STEP 1792 H=1792:00 +CPU= 33.441
  04:41:06 STEP 1793 H=1793:00 +CPU= 31.808
  04:41:53 STEP 1794 H=1794:00 +CPU= 45.295
  04:42:25 STEP 1795 H=1795:00 +CPU= 29.348
  04:42:57 STEP 1796 H=1796:00 +CPU= 29.923
  04:43:44 STEP 1797 H=1797:00 +CPU= 44.293
  04:44:15 STEP 1798 H=1798:00 +CPU= 29.831
  04:44:46 STEP 1799 H=1799:00 +CPU= 29.797
  04:45:40 STEP 1800 H=1800:00 +CPU= 51.650
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_74.zip
Uploading the intermediate file: upload_file_74.zip
  04:46:09 STEP 1801 H=1801:00 +CPU= 28.332
  04:46:39 STEP 1802 H=1802:00 +CPU= 29.292
  04:47:28 STEP 1803 H=1803:00 +CPU= 48.364
  04:48:05 STEP 1804 H=1804:00 +CPU= 34.451
  04:48:38 STEP 1805 H=1805:00 +CPU= 31.725
  04:49:31 STEP 1806 H=1806:00 +CPU= 50.570
  04:50:02 STEP 1807 H=1807:00 +CPU= 29.704
  04:50:33 STEP 1808 H=1808:00 +CPU= 29.466
  04:51:20 STEP 1809 H=1809:00 +CPU= 44.246
  04:51:50 STEP 1810 H=1810:00 +CPU= 29.748
  04:52:20 STEP 1811 H=1811:00 +CPU= 28.312
  04:53:10 STEP 1812 H=1812:00 +CPU= 48.971
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001812
  04:53:39 STEP 1813 H=1813:00 +CPU= 28.503
  04:54:10 STEP 1814 H=1814:00 +CPU= 29.326
  04:54:56 STEP 1815 H=1815:00 +CPU= 43.552
  04:55:29 STEP 1816 H=1816:00 +CPU= 31.388
  04:56:03 STEP 1817 H=1817:00 +CPU= 32.086
  04:56:56 STEP 1818 H=1818:00 +CPU= 49.826
  04:57:30 STEP 1819 H=1819:00 +CPU= 31.935
  04:58:04 STEP 1820 H=1820:00 +CPU= 33.578
  04:58:51 STEP 1821 H=1821:00 +CPU= 45.398
  04:59:21 STEP 1822 H=1822:00 +CPU= 29.262
  04:59:53 STEP 1823 H=1823:00 +CPU= 29.846
  05:00:45 STEP 1824 H=1824:00 +CPU= 50.490
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:01:16 STEP 1825 H=1825:00 +CPU= 29.002
  05:01:46 STEP 1826 H=1826:00 +CPU= 28.977
  05:02:32 STEP 1827 H=1827:00 +CPU= 44.446
  05:03:02 STEP 1828 H=1828:00 +CPU= 30.094
  05:03:32 STEP 1829 H=1829:00 +CPU= 28.156
  05:04:22 STEP 1830 H=1830:00 +CPU= 47.488
  05:04:54 STEP 1831 H=1831:00 +CPU= 31.137
  05:05:31 STEP 1832 H=1832:00 +CPU= 34.378
  05:06:24 STEP 1833 H=1833:00 +CPU= 49.562
  05:07:00 STEP 1834 H=1834:00 +CPU= 33.563
  05:07:33 STEP 1835 H=1835:00 +CPU= 31.102
  05:08:34 STEP 1836 H=1836:00 +CPU= 58.246
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001836
  05:09:08 STEP 1837 H=1837:00 +CPU= 33.189
  05:09:44 STEP 1838 H=1838:00 +CPU= 35.122
  05:10:44 STEP 1839 H=1839:00 +CPU= 56.976
  05:11:21 STEP 1840 H=1840:00 +CPU= 36.009
  05:11:57 STEP 1841 H=1841:00 +CPU= 33.174
  05:12:56 STEP 1842 H=1842:00 +CPU= 55.366
  05:13:32 STEP 1843 H=1843:00 +CPU= 35.307
  05:14:07 STEP 1844 H=1844:00 +CPU= 33.170
  05:15:01 STEP 1845 H=1845:00 +CPU= 52.253
  05:15:36 STEP 1846 H=1846:00 +CPU= 33.068
  05:16:10 STEP 1847 H=1847:00 +CPU= 32.148
  05:17:15 STEP 1848 H=1848:00 +CPU= 63.034
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_76.zip
Uploading the intermediate file: upload_file_76.zip
  05:17:53 STEP 1849 H=1849:00 +CPU= 36.122
  05:18:29 STEP 1850 H=1850:00 +CPU= 34.859
  05:19:24 STEP 1851 H=1851:00 +CPU= 53.022
  05:20:00 STEP 1852 H=1852:00 +CPU= 34.381
  05:20:31 STEP 1853 H=1853:00 +CPU= 30.592
  05:21:23 STEP 1854 H=1854:00 +CPU= 48.496
  05:21:55 STEP 1855 H=1855:00 +CPU= 31.525
  05:22:29 STEP 1856 H=1856:00 +CPU= 32.109
  05:23:20 STEP 1857 H=1857:00 +CPU= 49.034
  05:23:51 STEP 1858 H=1858:00 +CPU= 30.995
  05:24:25 STEP 1859 H=1859:00 +CPU= 31.816
  05:25:19 STEP 1860 H=1860:00 +CPU= 51.624
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001860
  05:25:52 STEP 1861 H=1861:00 +CPU= 31.869
  05:26:29 STEP 1862 H=1862:00 +CPU= 34.901
  05:27:33 STEP 1863 H=1863:00 +CPU= 61.458
  05:28:10 STEP 1864 H=1864:00 +CPU= 34.960
  05:28:46 STEP 1865 H=1865:00 +CPU= 34.176
  05:29:43 STEP 1866 H=1866:00 +CPU= 53.817
  05:30:26 STEP 1867 H=1867:00 +CPU= 40.421
  05:31:07 STEP 1868 H=1868:00 +CPU= 38.363
  05:32:18 STEP 1869 H=1869:00 +CPU= 65.235
  05:33:01 STEP 1870 H=1870:00 +CPU= 41.071
  05:33:53 STEP 1871 H=1871:00 +CPU= 51.715
  05:35:09 STEP 1872 H=1872:00 +CPU= 71.364
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_77.zip
Uploading the intermediate file: upload_file_77.zip
  05:35:45 STEP 1873 H=1873:00 +CPU= 34.402
  05:36:23 STEP 1874 H=1874:00 +CPU= 36.617
  05:37:31 STEP 1875 H=1875:00 +CPU= 58.459
  05:38:12 STEP 1876 H=1876:00 +CPU= 38.812
  05:38:49 STEP 1877 H=1877:00 +CPU= 34.987
  05:39:46 STEP 1878 H=1878:00 +CPU= 53.628
  05:40:20 STEP 1879 H=1879:00 +CPU= 32.834
  05:40:58 STEP 1880 H=1880:00 +CPU= 36.024
  05:41:57 STEP 1881 H=1881:00 +CPU= 56.885
  05:42:35 STEP 1882 H=1882:00 +CPU= 36.515
  05:43:13 STEP 1883 H=1883:00 +CPU= 35.529
  05:44:17 STEP 1884 H=1884:00 +CPU= 61.021
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001884
  05:44:51 STEP 1885 H=1885:00 +CPU= 32.147
  05:45:25 STEP 1886 H=1886:00 +CPU= 33.160
  05:46:20 STEP 1887 H=1887:00 +CPU= 51.781
  05:46:57 STEP 1888 H=1888:00 +CPU= 35.083
  05:47:29 STEP 1889 H=1889:00 +CPU= 31.760
  05:48:25 STEP 1890 H=1890:00 +CPU= 53.230
  05:48:58 STEP 1891 H=1891:00 +CPU= 31.626
  05:49:29 STEP 1892 H=1892:00 +CPU= 29.795
  05:50:18 STEP 1893 H=1893:00 +CPU= 46.833
  05:50:52 STEP 1894 H=1894:00 +CPU= 33.262
  05:51:30 STEP 1895 H=1895:00 +CPU= 36.275
  05:52:35 STEP 1896 H=1896:00 +CPU= 61.068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_78.zip
Uploading the intermediate file: upload_file_78.zip
  05:53:10 STEP 1897 H=1897:00 +CPU= 33.669
  05:53:49 STEP 1898 H=1898:00 +CPU= 36.940
  05:54:39 STEP 1899 H=1899:00 +CPU= 49.307
  05:55:16 STEP 1900 H=1900:00 +CPU= 34.746
  05:55:52 STEP 1901 H=1901:00 +CPU= 34.324
  05:56:51 STEP 1902 H=1902:00 +CPU= 55.935
  05:57:27 STEP 1903 H=1903:00 +CPU= 35.848
  05:58:06 STEP 1904 H=1904:00 +CPU= 36.682
  05:58:58 STEP 1905 H=1905:00 +CPU= 49.826
  05:59:30 STEP 1906 H=1906:00 +CPU= 30.844
  06:00:01 STEP 1907 H=1907:00 +CPU= 29.449
  06:01:01 STEP 1908 H=1908:00 +CPU= 58.957
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001908
  06:01:35 STEP 1909 H=1909:00 +CPU= 32.540
  06:02:08 STEP 1910 H=1910:00 +CPU= 31.083
  06:02:56 STEP 1911 H=1911:00 +CPU= 46.702
  06:03:27 STEP 1912 H=1912:00 +CPU= 29.632
  06:03:57 STEP 1913 H=1913:00 +CPU= 28.867
  06:04:44 STEP 1914 H=1914:00 +CPU= 46.260
  06:05:16 STEP 1915 H=1915:00 +CPU= 30.734
  06:05:48 STEP 1916 H=1916:00 +CPU= 30.031
  06:06:36 STEP 1917 H=1917:00 +CPU= 46.455
  06:07:08 STEP 1918 H=1918:00 +CPU= 30.477
  06:07:39 STEP 1919 H=1919:00 +CPU= 30.961
  06:08:35 STEP 1920 H=1920:00 +CPU= 52.848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_79.zip
Uploading the intermediate file: upload_file_79.zip
Uploading trickle at timestep: 6912000
  06:09:07 STEP 1921 H=1921:00 +CPU= 31.120
  06:09:40 STEP 1922 H=1922:00 +CPU= 31.534
  06:10:29 STEP 1923 H=1923:00 +CPU= 46.485
  06:11:00 STEP 1924 H=1924:00 +CPU= 30.599
  06:11:31 STEP 1925 H=1925:00 +CPU= 29.367
  06:12:17 STEP 1926 H=1926:00 +CPU= 44.373
  06:12:47 STEP 1927 H=1927:00 +CPU= 27.766
  06:13:16 STEP 1928 H=1928:00 +CPU= 28.530
  06:14:01 STEP 1929 H=1929:00 +CPU= 43.015
  06:14:30 STEP 1930 H=1930:00 +CPU= 28.830
  06:14:59 STEP 1931 H=1931:00 +CPU= 27.672
  06:15:48 STEP 1932 H=1932:00 +CPU= 47.373
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001932
  06:16:18 STEP 1933 H=1933:00 +CPU= 28.160
  06:16:47 STEP 1934 H=1934:00 +CPU= 27.944
  06:17:31 STEP 1935 H=1935:00 +CPU= 42.638
  06:18:00 STEP 1936 H=1936:00 +CPU= 27.897
  06:18:28 STEP 1937 H=1937:00 +CPU= 27.573
  06:19:15 STEP 1938 H=1938:00 +CPU= 44.793
  06:19:44 STEP 1939 H=1939:00 +CPU= 27.503
  06:20:13 STEP 1940 H=1940:00 +CPU= 28.178
  06:20:57 STEP 1941 H=1941:00 +CPU= 42.695
  06:21:27 STEP 1942 H=1942:00 +CPU= 28.337
  06:21:56 STEP 1943 H=1943:00 +CPU= 28.196
  06:22:48 STEP 1944 H=1944:00 +CPU= 49.631
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_80.zip
Uploading the intermediate file: upload_file_80.zip
  06:23:17 STEP 1945 H=1945:00 +CPU= 27.671
  06:23:46 STEP 1946 H=1946:00 +CPU= 28.431
  06:24:29 STEP 1947 H=1947:00 +CPU= 42.545
  06:24:59 STEP 1948 H=1948:00 +CPU= 28.418
  06:25:27 STEP 1949 H=1949:00 +CPU= 27.481
  06:26:13 STEP 1950 H=1950:00 +CPU= 43.472
  06:26:42 STEP 1951 H=1951:00 +CPU= 27.863
  06:27:11 STEP 1952 H=1952:00 +CPU= 28.368
  06:27:56 STEP 1953 H=1953:00 +CPU= 43.870
  06:28:26 STEP 1954 H=1954:00 +CPU= 28.650
  06:28:54 STEP 1955 H=1955:00 +CPU= 27.631
  06:29:44 STEP 1956 H=1956:00 +CPU= 47.351
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001956
  06:30:14 STEP 1957 H=1957:00 +CPU= 28.172
  06:30:43 STEP 1958 H=1958:00 +CPU= 28.212
  06:31:27 STEP 1959 H=1959:00 +CPU= 42.784
  06:31:58 STEP 1960 H=1960:00 +CPU= 29.298
  06:32:26 STEP 1961 H=1961:00 +CPU= 27.543
  06:33:13 STEP 1962 H=1962:00 +CPU= 44.444
  06:33:42 STEP 1963 H=1963:00 +CPU= 27.536
  06:34:12 STEP 1964 H=1964:00 +CPU= 28.186
  06:34:57 STEP 1965 H=1965:00 +CPU= 43.375
  06:35:27 STEP 1966 H=1966:00 +CPU= 28.929
  06:35:57 STEP 1967 H=1967:00 +CPU= 28.381
  06:36:49 STEP 1968 H=1968:00 +CPU= 49.943
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_81.zip
Uploading the intermediate file: upload_file_81.zip
  06:37:19 STEP 1969 H=1969:00 +CPU= 27.846
  06:37:48 STEP 1970 H=1970:00 +CPU= 28.293
  06:38:33 STEP 1971 H=1971:00 +CPU= 42.832
  06:39:03 STEP 1972 H=1972:00 +CPU= 29.032
  06:39:31 STEP 1973 H=1973:00 +CPU= 27.808
  06:40:17 STEP 1974 H=1974:00 +CPU= 43.234
  06:40:46 STEP 1975 H=1975:00 +CPU= 27.802
  06:41:15 STEP 1976 H=1976:00 +CPU= 28.348
  06:42:01 STEP 1977 H=1977:00 +CPU= 43.910
  06:42:31 STEP 1978 H=1978:00 +CPU= 28.852
  06:43:00 STEP 1979 H=1979:00 +CPU= 27.846
  06:43:52 STEP 1980 H=1980:00 +CPU= 47.440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001980
  06:44:21 STEP 1981 H=1981:00 +CPU= 28.016
  06:44:50 STEP 1982 H=1982:00 +CPU= 28.183
  06:45:36 STEP 1983 H=1983:00 +CPU= 43.666
  06:46:06 STEP 1984 H=1984:00 +CPU= 28.514
  06:46:34 STEP 1985 H=1985:00 +CPU= 26.966
  06:47:19 STEP 1986 H=1986:00 +CPU= 43.780
  06:47:48 STEP 1987 H=1987:00 +CPU= 27.374
  06:48:18 STEP 1988 H=1988:00 +CPU= 28.502
  06:49:03 STEP 1989 H=1989:00 +CPU= 43.246
  06:49:33 STEP 1990 H=1990:00 +CPU= 28.536
  06:50:03 STEP 1991 H=1991:00 +CPU= 28.130
  06:50:55 STEP 1992 H=1992:00 +CPU= 51.003
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_82.zip
Uploading the intermediate file: upload_file_82.zip
  06:51:23 STEP 1993 H=1993:00 +CPU= 27.740
  06:51:53 STEP 1994 H=1994:00 +CPU= 28.193
  06:52:38 STEP 1995 H=1995:00 +CPU= 42.337
  06:53:08 STEP 1996 H=1996:00 +CPU= 29.141
  06:53:37 STEP 1997 H=1997:00 +CPU= 27.412
  06:54:22 STEP 1998 H=1998:00 +CPU= 44.076
  06:54:51 STEP 1999 H=1999:00 +CPU= 28.194
  06:55:21 STEP 2000 H=2000:00 +CPU= 28.702
  06:56:06 STEP 2001 H=2001:00 +CPU= 43.005
  06:56:37 STEP 2002 H=2002:00 +CPU= 28.942
  06:57:05 STEP 2003 H=2003:00 +CPU= 27.313
  06:57:55 STEP 2004 H=2004:00 +CPU= 47.841
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002004
  06:58:23 STEP 2005 H=2005:00 +CPU= 27.724
  06:58:53 STEP 2006 H=2006:00 +CPU= 28.233
  06:59:39 STEP 2007 H=2007:00 +CPU= 43.007
  07:00:11 STEP 2008 H=2008:00 +CPU= 28.797
  07:00:41 STEP 2009 H=2009:00 +CPU= 27.418
  07:01:28 STEP 2010 H=2010:00 +CPU= 44.085
  07:01:55 STEP 2011 H=2011:00 +CPU= 27.385
  07:02:24 STEP 2012 H=2012:00 +CPU= 28.109
  07:03:10 STEP 2013 H=2013:00 +CPU= 43.824
  07:03:39 STEP 2014 H=2014:00 +CPU= 28.228
  07:04:10 STEP 2015 H=2015:00 +CPU= 28.491
  07:05:03 STEP 2016 H=2016:00 +CPU= 50.712
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_83.zip
Uploading the intermediate file: upload_file_83.zip
  07:05:33 STEP 2017 H=2017:00 +CPU= 27.982
  07:06:03 STEP 2018 H=2018:00 +CPU= 28.238
  07:06:47 STEP 2019 H=2019:00 +CPU= 43.000
  07:07:17 STEP 2020 H=2020:00 +CPU= 29.152
  07:07:46 STEP 2021 H=2021:00 +CPU= 27.486
  07:08:33 STEP 2022 H=2022:00 +CPU= 44.083
  07:09:02 STEP 2023 H=2023:00 +CPU= 27.781
  07:09:31 STEP 2024 H=2024:00 +CPU= 28.084
  07:10:15 STEP 2025 H=2025:00 +CPU= 42.966
  07:10:44 STEP 2026 H=2026:00 +CPU= 28.445
  07:11:13 STEP 2027 H=2027:00 +CPU= 27.259
  07:12:02 STEP 2028 H=2028:00 +CPU= 47.651
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002028
  07:12:32 STEP 2029 H=2029:00 +CPU= 28.264
  07:13:03 STEP 2030 H=2030:00 +CPU= 28.496
  07:13:47 STEP 2031 H=2031:00 +CPU= 42.796
  07:14:18 STEP 2032 H=2032:00 +CPU= 28.952
  07:14:46 STEP 2033 H=2033:00 +CPU= 26.995
  07:15:31 STEP 2034 H=2034:00 +CPU= 43.504
  07:16:00 STEP 2035 H=2035:00 +CPU= 27.874
  07:16:29 STEP 2036 H=2036:00 +CPU= 28.212
  07:17:13 STEP 2037 H=2037:00 +CPU= 42.856
  07:17:43 STEP 2038 H=2038:00 +CPU= 28.406
  07:18:12 STEP 2039 H=2039:00 +CPU= 28.321
  07:19:05 STEP 2040 H=2040:00 +CPU= 50.289
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_84.zip
Uploading the intermediate file: upload_file_84.zip
  07:19:33 STEP 2041 H=2041:00 +CPU= 27.405
  07:20:02 STEP 2042 H=2042:00 +CPU= 28.243
  07:20:46 STEP 2043 H=2043:00 +CPU= 42.485
  07:21:16 STEP 2044 H=2044:00 +CPU= 28.740
  07:21:44 STEP 2045 H=2045:00 +CPU= 27.442
  07:22:31 STEP 2046 H=2046:00 +CPU= 44.104
  07:23:01 STEP 2047 H=2047:00 +CPU= 28.424
  07:23:30 STEP 2048 H=2048:00 +CPU= 28.586
  07:24:14 STEP 2049 H=2049:00 +CPU= 43.144
  07:24:45 STEP 2050 H=2050:00 +CPU= 29.413
  07:25:14 STEP 2051 H=2051:00 +CPU= 27.483
  07:26:05 STEP 2052 H=2052:00 +CPU= 48.025
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002052
  07:26:33 STEP 2053 H=2053:00 +CPU= 27.396
  07:27:03 STEP 2054 H=2054:00 +CPU= 28.427
  07:27:49 STEP 2055 H=2055:00 +CPU= 43.153
  07:28:17 STEP 2056 H=2056:00 +CPU= 28.122
  07:28:46 STEP 2057 H=2057:00 +CPU= 27.849
  07:29:33 STEP 2058 H=2058:00 +CPU= 44.458
  07:30:02 STEP 2059 H=2059:00 +CPU= 28.503
  07:30:31 STEP 2060 H=2060:00 +CPU= 28.410
  07:31:16 STEP 2061 H=2061:00 +CPU= 43.265
  07:31:46 STEP 2062 H=2062:00 +CPU= 28.343
  07:32:15 STEP 2063 H=2063:00 +CPU= 27.988
  07:33:07 STEP 2064 H=2064:00 +CPU= 50.055
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_85.zip
Uploading the intermediate file: upload_file_85.zip
  07:33:35 STEP 2065 H=2065:00 +CPU= 27.848
  07:34:05 STEP 2066 H=2066:00 +CPU= 28.082
  07:34:49 STEP 2067 H=2067:00 +CPU= 42.737
  07:35:20 STEP 2068 H=2068:00 +CPU= 28.972
  07:35:49 STEP 2069 H=2069:00 +CPU= 27.456
  07:36:34 STEP 2070 H=2070:00 +CPU= 43.569
  07:37:03 STEP 2071 H=2071:00 +CPU= 28.050
  07:37:32 STEP 2072 H=2072:00 +CPU= 28.294
  07:38:17 STEP 2073 H=2073:00 +CPU= 42.763
  07:38:47 STEP 2074 H=2074:00 +CPU= 28.739
  07:39:16 STEP 2075 H=2075:00 +CPU= 27.477
  07:40:06 STEP 2076 H=2076:00 +CPU= 47.822
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002076
  07:40:35 STEP 2077 H=2077:00 +CPU= 27.847
  07:41:04 STEP 2078 H=2078:00 +CPU= 28.098
  07:41:49 STEP 2079 H=2079:00 +CPU= 42.898
  07:42:19 STEP 2080 H=2080:00 +CPU= 28.639
  07:42:47 STEP 2081 H=2081:00 +CPU= 27.542
  07:43:32 STEP 2082 H=2082:00 +CPU= 43.553
  07:44:00 STEP 2083 H=2083:00 +CPU= 27.485
  07:44:31 STEP 2084 H=2084:00 +CPU= 28.427
  07:45:18 STEP 2085 H=2085:00 +CPU= 42.936
  07:45:48 STEP 2086 H=2086:00 +CPU= 28.865
  07:46:17 STEP 2087 H=2087:00 +CPU= 28.040
  07:47:10 STEP 2088 H=2088:00 +CPU= 50.112
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_86.zip
Uploading the intermediate file: upload_file_86.zip
  07:47:39 STEP 2089 H=2089:00 +CPU= 28.125
  07:48:08 STEP 2090 H=2090:00 +CPU= 27.886
  07:48:53 STEP 2091 H=2091:00 +CPU= 42.721
  07:49:23 STEP 2092 H=2092:00 +CPU= 28.108
  07:49:52 STEP 2093 H=2093:00 +CPU= 27.220
  07:50:39 STEP 2094 H=2094:00 +CPU= 44.179
  07:51:08 STEP 2095 H=2095:00 +CPU= 27.570
  07:51:38 STEP 2096 H=2096:00 +CPU= 28.333
  07:52:22 STEP 2097 H=2097:00 +CPU= 42.573
  07:52:52 STEP 2098 H=2098:00 +CPU= 29.253
  07:53:21 STEP 2099 H=2099:00 +CPU= 27.315
  07:54:08 STEP 2100 H=2100:00 +CPU= 47.025
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002100
  07:54:39 STEP 2101 H=2101:00 +CPU= 28.441
  07:55:08 STEP 2102 H=2102:00 +CPU= 28.086
  07:55:53 STEP 2103 H=2103:00 +CPU= 42.444
  07:56:22 STEP 2104 H=2104:00 +CPU= 28.535
  07:56:51 STEP 2105 H=2105:00 +CPU= 27.516
  07:57:37 STEP 2106 H=2106:00 +CPU= 44.463
  07:58:06 STEP 2107 H=2107:00 +CPU= 27.602
  07:58:34 STEP 2108 H=2108:00 +CPU= 27.934
  07:59:19 STEP 2109 H=2109:00 +CPU= 42.595
  07:59:49 STEP 2110 H=2110:00 +CPU= 28.436
  08:00:18 STEP 2111 H=2111:00 +CPU= 28.144
  08:01:09 STEP 2112 H=2112:00 +CPU= 49.161
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_87.zip
Uploading the intermediate file: upload_file_87.zip
  08:01:39 STEP 2113 H=2113:00 +CPU= 27.719
  08:02:08 STEP 2114 H=2114:00 +CPU= 27.778
  08:02:52 STEP 2115 H=2115:00 +CPU= 42.964
  08:03:22 STEP 2116 H=2116:00 +CPU= 28.688
  08:03:51 STEP 2117 H=2117:00 +CPU= 27.499
  08:04:37 STEP 2118 H=2118:00 +CPU= 43.048
  08:05:06 STEP 2119 H=2119:00 +CPU= 28.171
  08:05:35 STEP 2120 H=2120:00 +CPU= 28.305
  08:06:20 STEP 2121 H=2121:00 +CPU= 42.912
  08:06:49 STEP 2122 H=2122:00 +CPU= 28.132
  08:07:17 STEP 2123 H=2123:00 +CPU= 27.479
  08:08:06 STEP 2124 H=2124:00 +CPU= 47.202
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002124
  08:08:35 STEP 2125 H=2125:00 +CPU= 27.524
  08:09:04 STEP 2126 H=2126:00 +CPU= 28.184
  08:09:48 STEP 2127 H=2127:00 +CPU= 43.220
  08:10:18 STEP 2128 H=2128:00 +CPU= 28.789
  08:10:46 STEP 2129 H=2129:00 +CPU= 27.450
  08:11:32 STEP 2130 H=2130:00 +CPU= 44.002
  08:12:01 STEP 2131 H=2131:00 +CPU= 27.685
  08:12:31 STEP 2132 H=2132:00 +CPU= 28.153
  08:13:16 STEP 2133 H=2133:00 +CPU= 42.744
  08:13:46 STEP 2134 H=2134:00 +CPU= 28.166
  08:14:14 STEP 2135 H=2135:00 +CPU= 27.792
  08:15:07 STEP 2136 H=2136:00 +CPU= 50.300
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_88.zip
Uploading the intermediate file: upload_file_88.zip
  08:15:35 STEP 2137 H=2137:00 +CPU= 27.104
  08:16:04 STEP 2138 H=2138:00 +CPU= 28.106
  08:16:49 STEP 2139 H=2139:00 +CPU= 42.761
  08:17:19 STEP 2140 H=2140:00 +CPU= 29.138
  08:17:47 STEP 2141 H=2141:00 +CPU= 26.902
  08:18:35 STEP 2142 H=2142:00 +CPU= 44.112
  08:19:05 STEP 2143 H=2143:00 +CPU= 27.569
  08:19:34 STEP 2144 H=2144:00 +CPU= 28.000
  08:20:18 STEP 2145 H=2145:00 +CPU= 42.743
  08:20:48 STEP 2146 H=2146:00 +CPU= 28.402
  08:21:16 STEP 2147 H=2147:00 +CPU= 26.948
  08:22:04 STEP 2148 H=2148:00 +CPU= 47.156
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002148
  08:22:34 STEP 2149 H=2149:00 +CPU= 28.358
  08:23:04 STEP 2150 H=2150:00 +CPU= 28.449
  08:23:50 STEP 2151 H=2151:00 +CPU= 43.330
  08:24:19 STEP 2152 H=2152:00 +CPU= 28.572
  08:24:47 STEP 2153 H=2153:00 +CPU= 27.153
  08:25:33 STEP 2154 H=2154:00 +CPU= 43.859
  08:26:02 STEP 2155 H=2155:00 +CPU= 27.705
  08:26:31 STEP 2156 H=2156:00 +CPU= 27.881
  08:27:15 STEP 2157 H=2157:00 +CPU= 42.792
  08:27:44 STEP 2158 H=2158:00 +CPU= 28.779
  08:28:13 STEP 2159 H=2159:00 +CPU= 27.975
  08:29:06 STEP 2160 H=2160:00 +CPU= 50.096
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_89.zip
Uploading the intermediate file: upload_file_89.zip
Uploading trickle at timestep: 7776000
  08:29:34 STEP 2161 H=2161:00 +CPU= 27.490
  08:30:03 STEP 2162 H=2162:00 +CPU= 27.989
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083008:1672990208:18347.280] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4293MB (heap), 4416MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4293 MB (maxheap), 4416 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            RADDRV 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             RADINTG 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-RADINTG-INPUT LOOP3  (1207) 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083009:1672990209:18347.471] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAIDDIOBS 
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083010:1672990210:18348.471] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:14372:14372] [20230106:083010:1672990210:18348.522] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000012CAABD  laiddiobs_                 77  laiddiobs.F90
oifs_43r3_model.e  00000000012C0E8E  radintg_                 1051  radintg.F90
oifs_43r3_model.e  0000000001270124  raddrv_                   537  raddrv.F90
oifs_43r3_model.e  0000000000BFD839  gp_model_                 482  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0234
(argv4) batchid: 963
(argv5) wuid: 12179878
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2f46728cb1dad8c4548f82cf708ea11b to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0234_1994050100_123_963_12179878.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0234_1994050100_123_963_12179878.zip
ic_ancil_file: ic_ancil_12179878
ifsdata_file: ifsdata_12179878
climate_data_file: clim_data_12179878
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3a03b987c75139e61264c810a4c604dc to: /var/lib/boinc-client/slots/4/ic_ancil_12179878.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12179878.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0aa549896452496ae4dc4a9dcc966b06 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12179878.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12179878.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12179878.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12179878.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12179878.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12179878.xml
-- Model is restarting --
Adjusting last_iter, 2163, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12179878.xml
last_cpu_time: 77315
upload_file_number: 90
last_iter: 2159
last_upload: 7776000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767
The child process has been launched with process id: 1747
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:WRZTELEBIMWSKL01:1:1:1747:1747] [20230106:083251:1672990371:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:34:42 STEP 2160 H=2160:00 +CPU= 74.625
  08:35:20 STEP 2161 H=2161:00 +CPU= 38.035
  08:36:02 STEP 2162 H=2162:00 +CPU= 41.682
  08:37:05 STEP 2163 H=2163:00 +CPU= 63.374
  08:37:50 STEP 2164 H=2164:00 +CPU= 44.256
  08:38:32 STEP 2165 H=2165:00 +CPU= 42.310
  08:39:33 STEP 2166 H=2166:00 +CPU= 60.418
  08:40:27 STEP 2167 H=2167:00 +CPU= 54.007
  08:41:22 STEP 2168 H=2168:00 +CPU= 55.444
  08:42:31 STEP 2169 H=2169:00 +CPU= 68.709
  08:43:15 STEP 2170 H=2170:00 +CPU= 43.947
  08:44:00 STEP 2171 H=2171:00 +CPU= 44.547
  08:45:05 STEP 2172 H=2172:00 +CPU= 65.029
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002172
  08:45:48 STEP 2173 H=2173:00 +CPU= 42.773
  08:46:31 STEP 2174 H=2174:00 +CPU= 43.023
  08:47:34 STEP 2175 H=2175:00 +CPU= 62.382
  08:48:16 STEP 2176 H=2176:00 +CPU= 42.247
  08:48:58 STEP 2177 H=2177:00 +CPU= 42.058
  08:50:11 STEP 2178 H=2178:00 +CPU= 72.323
  08:50:48 STEP 2179 H=2179:00 +CPU= 37.289
  08:51:28 STEP 2180 H=2180:00 +CPU= 39.830
  08:52:24 STEP 2181 H=2181:00 +CPU= 55.828
  08:53:03 STEP 2182 H=2182:00 +CPU= 38.979
  08:53:48 STEP 2183 H=2183:00 +CPU= 45.403
  08:54:54 STEP 2184 H=2184:00 +CPU= 65.750
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179878/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0234_1994050100_123_963_12179878_0_r1048664767_90.zip
Uploading the intermediate file: upload_file_90.zip
  08:55:33 STEP 2185 H=2185:00 +CPU= 38.276
  08:56:13 STEP 2186 H=2186:00 +CPU= 39.802
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:10:07 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 7,776,000 77,256 0.0099
09 May 2023 16:10:07 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 7,776,000 77,256 0.0099
09 May 2023 16:09:58 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 6,912,000 69,181 0.0100
09 May 2023 16:09:25 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 5,270,400 51,702 0.0098
09 May 2023 16:09:15 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 4,406,400 43,158 0.0098
09 May 2023 16:08:49 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 2,937,600 27,979 0.0095
09 May 2023 16:08:37 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 1,728,000 16,006 0.0093
09 May 2023 16:08:30 1533065 22264812 oifs_43r3_ps_0234_1994050100_123_963_12179878_0 864,000 7,708 0.0089


©2024 climateprediction.net