climateprediction.net home page
Task 22286148

Task 22286148

Name oifs_43r3_ps_0295_2014050100_123_983_12199939_0
Workunit 12199939
Created 24 Dec 2022, 9:48:34 UTC
Sent 9 Jan 2023, 5:42:54 UTC
Report deadline 8 Feb 2023, 5:42:54 UTC
Received 10 Jan 2023, 2:54:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1502085
Run time 13 hours 37 min 24 sec
CPU time 12 hours 42 min 25 sec
Validate state Invalid
Credit 9,192.96
Device peak FLOPS 6.43 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,767.49 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,323.85 MB
Stderr
<core_client_version>7.20.4</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
Ahq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_70.zip
Uploading the intermediate file: upload_file_70.zip
  16:53:24 STEP 1705 H=1705:00 +CPU= 16.826
  16:53:41 STEP 1706 H=1706:00 +CPU= 17.613
  16:54:07 STEP 1707 H=1707:00 +CPU= 25.587
  16:54:25 STEP 1708 H=1708:00 +CPU= 17.446
  16:54:41 STEP 1709 H=1709:00 +CPU= 16.318
  16:55:09 STEP 1710 H=1710:00 +CPU= 28.161
  16:55:27 STEP 1711 H=1711:00 +CPU= 18.008
  16:55:45 STEP 1712 H=1712:00 +CPU= 17.547
  16:56:13 STEP 1713 H=1713:00 +CPU= 27.497
  16:56:31 STEP 1714 H=1714:00 +CPU= 17.819
  16:56:46 STEP 1715 H=1715:00 +CPU= 15.105
  16:57:14 STEP 1716 H=1716:00 +CPU= 27.796
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001716
  16:57:31 STEP 1717 H=1717:00 +CPU= 16.817
  16:57:48 STEP 1718 H=1718:00 +CPU= 17.083
  16:58:15 STEP 1719 H=1719:00 +CPU= 27.002
  16:58:33 STEP 1720 H=1720:00 +CPU= 17.592
  16:58:51 STEP 1721 H=1721:00 +CPU= 17.951
  16:59:18 STEP 1722 H=1722:00 +CPU= 26.854
  16:59:35 STEP 1723 H=1723:00 +CPU= 16.803
  16:59:53 STEP 1724 H=1724:00 +CPU= 17.456
  17:00:20 STEP 1725 H=1725:00 +CPU= 26.582
  17:00:37 STEP 1726 H=1726:00 +CPU= 17.218
  17:00:54 STEP 1727 H=1727:00 +CPU= 16.992
  17:01:26 STEP 1728 H=1728:00 +CPU= 31.106
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_71.zip
Uploading the intermediate file: upload_file_71.zip
  17:01:44 STEP 1729 H=1729:00 +CPU= 17.936
  17:02:01 STEP 1730 H=1730:00 +CPU= 17.385
  17:02:28 STEP 1731 H=1731:00 +CPU= 26.720
  17:02:45 STEP 1732 H=1732:00 +CPU= 17.061
  17:03:03 STEP 1733 H=1733:00 +CPU= 17.071
  17:03:29 STEP 1734 H=1734:00 +CPU= 25.936
  17:03:46 STEP 1735 H=1735:00 +CPU= 17.268
  17:04:03 STEP 1736 H=1736:00 +CPU= 16.552
  17:04:29 STEP 1737 H=1737:00 +CPU= 25.837
  17:04:47 STEP 1738 H=1738:00 +CPU= 18.126
  17:05:05 STEP 1739 H=1739:00 +CPU= 17.262
  17:05:35 STEP 1740 H=1740:00 +CPU= 30.765
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001740
  17:05:53 STEP 1741 H=1741:00 +CPU= 17.544
  17:06:11 STEP 1742 H=1742:00 +CPU= 17.271
  17:06:37 STEP 1743 H=1743:00 +CPU= 26.587
  17:06:55 STEP 1744 H=1744:00 +CPU= 17.675
  17:07:12 STEP 1745 H=1745:00 +CPU= 16.785
  17:07:40 STEP 1746 H=1746:00 +CPU= 27.708
  17:07:57 STEP 1747 H=1747:00 +CPU= 16.643
  17:08:14 STEP 1748 H=1748:00 +CPU= 17.259
  17:08:41 STEP 1749 H=1749:00 +CPU= 26.849
  17:08:58 STEP 1750 H=1750:00 +CPU= 17.156
  17:09:15 STEP 1751 H=1751:00 +CPU= 16.622
  17:09:47 STEP 1752 H=1752:00 +CPU= 32.073
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_72.zip
Uploading the intermediate file: upload_file_72.zip
  17:10:05 STEP 1753 H=1753:00 +CPU= 17.260
  17:10:23 STEP 1754 H=1754:00 +CPU= 17.429
  17:10:49 STEP 1755 H=1755:00 +CPU= 26.761
  17:11:07 STEP 1756 H=1756:00 +CPU= 17.016
  17:11:24 STEP 1757 H=1757:00 +CPU= 16.888
  17:11:50 STEP 1758 H=1758:00 +CPU= 26.365
  17:12:07 STEP 1759 H=1759:00 +CPU= 16.607
  17:12:24 STEP 1760 H=1760:00 +CPU= 17.000
  17:12:49 STEP 1761 H=1761:00 +CPU= 24.431
  17:13:05 STEP 1762 H=1762:00 +CPU= 16.122
  17:13:21 STEP 1763 H=1763:00 +CPU= 16.078
  17:13:49 STEP 1764 H=1764:00 +CPU= 28.076
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001764
  17:14:06 STEP 1765 H=1765:00 +CPU= 16.571
  17:14:23 STEP 1766 H=1766:00 +CPU= 16.976
  17:14:50 STEP 1767 H=1767:00 +CPU= 26.312
  17:15:06 STEP 1768 H=1768:00 +CPU= 15.745
  17:15:22 STEP 1769 H=1769:00 +CPU= 16.042
  17:15:46 STEP 1770 H=1770:00 +CPU= 24.619
  17:16:02 STEP 1771 H=1771:00 +CPU= 15.728
  17:16:19 STEP 1772 H=1772:00 +CPU= 16.603
  17:16:45 STEP 1773 H=1773:00 +CPU= 25.382
  17:17:01 STEP 1774 H=1774:00 +CPU= 16.463
  17:17:18 STEP 1775 H=1775:00 +CPU= 16.993
  17:17:49 STEP 1776 H=1776:00 +CPU= 30.175
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_73.zip
Uploading the intermediate file: upload_file_73.zip
  17:18:05 STEP 1777 H=1777:00 +CPU= 15.878
  17:18:20 STEP 1778 H=1778:00 +CPU= 14.877
  17:18:44 STEP 1779 H=1779:00 +CPU= 24.306
  17:19:01 STEP 1780 H=1780:00 +CPU= 16.533
  17:19:16 STEP 1781 H=1781:00 +CPU= 14.835
  17:19:40 STEP 1782 H=1782:00 +CPU= 24.439
  17:19:57 STEP 1783 H=1783:00 +CPU= 16.013
  17:20:13 STEP 1784 H=1784:00 +CPU= 16.219
  17:20:36 STEP 1785 H=1785:00 +CPU= 23.065
  17:20:52 STEP 1786 H=1786:00 +CPU= 15.521
  17:21:08 STEP 1787 H=1787:00 +CPU= 15.698
  17:21:36 STEP 1788 H=1788:00 +CPU= 27.803
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001788
  17:21:51 STEP 1789 H=1789:00 +CPU= 15.794
  17:22:07 STEP 1790 H=1790:00 +CPU= 15.540
  17:22:31 STEP 1791 H=1791:00 +CPU= 24.066
  17:22:50 STEP 1792 H=1792:00 +CPU= 18.368
  17:23:06 STEP 1793 H=1793:00 +CPU= 15.964
  17:23:31 STEP 1794 H=1794:00 +CPU= 24.707
  17:23:47 STEP 1795 H=1795:00 +CPU= 16.523
  17:24:04 STEP 1796 H=1796:00 +CPU= 16.058
  17:24:29 STEP 1797 H=1797:00 +CPU= 25.289
  17:24:45 STEP 1798 H=1798:00 +CPU= 16.044
  17:25:03 STEP 1799 H=1799:00 +CPU= 17.143
  17:25:34 STEP 1800 H=1800:00 +CPU= 31.358
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_74.zip
Uploading the intermediate file: upload_file_74.zip
  17:25:51 STEP 1801 H=1801:00 +CPU= 16.345
  17:26:08 STEP 1802 H=1802:00 +CPU= 16.716
  17:26:33 STEP 1803 H=1803:00 +CPU= 25.460
  17:26:50 STEP 1804 H=1804:00 +CPU= 16.546
  17:27:06 STEP 1805 H=1805:00 +CPU= 15.616
  17:27:32 STEP 1806 H=1806:00 +CPU= 25.632
  17:27:47 STEP 1807 H=1807:00 +CPU= 15.435
  17:28:03 STEP 1808 H=1808:00 +CPU= 15.560
  17:28:27 STEP 1809 H=1809:00 +CPU= 24.528
  17:28:42 STEP 1810 H=1810:00 +CPU= 14.618
  17:28:57 STEP 1811 H=1811:00 +CPU= 14.494
  17:29:22 STEP 1812 H=1812:00 +CPU= 25.357
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001812
  17:29:36 STEP 1813 H=1813:00 +CPU= 13.448
  17:29:51 STEP 1814 H=1814:00 +CPU= 14.988
  17:30:14 STEP 1815 H=1815:00 +CPU= 23.277
  17:30:29 STEP 1816 H=1816:00 +CPU= 14.829
  17:30:44 STEP 1817 H=1817:00 +CPU= 14.639
  17:31:07 STEP 1818 H=1818:00 +CPU= 23.523
  17:31:22 STEP 1819 H=1819:00 +CPU= 14.468
  17:31:36 STEP 1820 H=1820:00 +CPU= 13.640
  17:31:58 STEP 1821 H=1821:00 +CPU= 22.388
  17:32:13 STEP 1822 H=1822:00 +CPU= 14.885
  17:32:28 STEP 1823 H=1823:00 +CPU= 14.603
  17:32:55 STEP 1824 H=1824:00 +CPU= 26.769
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_75.zip
Uploading the intermediate file: upload_file_75.zip
  17:33:10 STEP 1825 H=1825:00 +CPU= 14.784
Uploading trickle at timestep: 6566400
  17:33:24 STEP 1826 H=1826:00 +CPU= 14.227
  17:33:46 STEP 1827 H=1827:00 +CPU= 22.134
  17:34:02 STEP 1828 H=1828:00 +CPU= 15.813
  17:34:16 STEP 1829 H=1829:00 +CPU= 13.982
  17:34:40 STEP 1830 H=1830:00 +CPU= 24.098
  17:34:55 STEP 1831 H=1831:00 +CPU= 14.613
  17:35:10 STEP 1832 H=1832:00 +CPU= 14.896
  17:35:33 STEP 1833 H=1833:00 +CPU= 22.980
  17:35:50 STEP 1834 H=1834:00 +CPU= 16.461
  17:36:06 STEP 1835 H=1835:00 +CPU= 16.022
  17:36:33 STEP 1836 H=1836:00 +CPU= 27.132
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001836
  17:36:48 STEP 1837 H=1837:00 +CPU= 14.215
  17:37:04 STEP 1838 H=1838:00 +CPU= 15.848
  17:37:27 STEP 1839 H=1839:00 +CPU= 23.556
  17:37:43 STEP 1840 H=1840:00 +CPU= 16.021
  17:37:58 STEP 1841 H=1841:00 +CPU= 14.889
  17:38:21 STEP 1842 H=1842:00 +CPU= 22.859
  17:38:37 STEP 1843 H=1843:00 +CPU= 15.251
  17:38:52 STEP 1844 H=1844:00 +CPU= 15.364
  17:39:17 STEP 1845 H=1845:00 +CPU= 24.862
  17:39:34 STEP 1846 H=1846:00 +CPU= 16.288
  17:39:50 STEP 1847 H=1847:00 +CPU= 16.326
  17:40:20 STEP 1848 H=1848:00 +CPU= 29.367
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_76.zip
Uploading the intermediate file: upload_file_76.zip
  17:40:36 STEP 1849 H=1849:00 +CPU= 15.776
  17:40:52 STEP 1850 H=1850:00 +CPU= 16.254
  17:41:17 STEP 1851 H=1851:00 +CPU= 24.590
  17:41:33 STEP 1852 H=1852:00 +CPU= 16.316
  17:41:49 STEP 1853 H=1853:00 +CPU= 15.895
  17:42:15 STEP 1854 H=1854:00 +CPU= 24.986
  17:42:30 STEP 1855 H=1855:00 +CPU= 14.942
  17:42:46 STEP 1856 H=1856:00 +CPU= 16.411
  17:43:12 STEP 1857 H=1857:00 +CPU= 25.283
  17:43:28 STEP 1858 H=1858:00 +CPU= 15.955
  17:43:44 STEP 1859 H=1859:00 +CPU= 16.141
  17:44:12 STEP 1860 H=1860:00 +CPU= 27.948
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001860
  17:44:28 STEP 1861 H=1861:00 +CPU= 15.935
  17:44:45 STEP 1862 H=1862:00 +CPU= 16.149
  17:45:11 STEP 1863 H=1863:00 +CPU= 25.833
  17:45:27 STEP 1864 H=1864:00 +CPU= 16.487
  17:45:43 STEP 1865 H=1865:00 +CPU= 15.314
  17:46:08 STEP 1866 H=1866:00 +CPU= 25.557
  17:46:23 STEP 1867 H=1867:00 +CPU= 14.257
  17:46:39 STEP 1868 H=1868:00 +CPU= 15.775
  17:47:03 STEP 1869 H=1869:00 +CPU= 24.459
  17:47:20 STEP 1870 H=1870:00 +CPU= 16.132
  17:47:36 STEP 1871 H=1871:00 +CPU= 16.656
  17:48:06 STEP 1872 H=1872:00 +CPU= 28.878
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_77.zip
Uploading the intermediate file: upload_file_77.zip
  17:48:20 STEP 1873 H=1873:00 +CPU= 14.417
  17:48:36 STEP 1874 H=1874:00 +CPU= 15.905
  17:49:01 STEP 1875 H=1875:00 +CPU= 24.422
  17:49:17 STEP 1876 H=1876:00 +CPU= 16.173
  17:49:32 STEP 1877 H=1877:00 +CPU= 14.918
  17:49:58 STEP 1878 H=1878:00 +CPU= 25.151
  17:50:14 STEP 1879 H=1879:00 +CPU= 15.835
  17:50:30 STEP 1880 H=1880:00 +CPU= 16.372
  17:50:55 STEP 1881 H=1881:00 +CPU= 25.030
  17:51:12 STEP 1882 H=1882:00 +CPU= 16.015
  17:51:27 STEP 1883 H=1883:00 +CPU= 15.437
  17:51:55 STEP 1884 H=1884:00 +CPU= 27.479
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001884
  17:52:10 STEP 1885 H=1885:00 +CPU= 15.224
  17:52:25 STEP 1886 H=1886:00 +CPU= 15.009
  17:52:51 STEP 1887 H=1887:00 +CPU= 25.750
  17:53:07 STEP 1888 H=1888:00 +CPU= 15.430
  17:53:22 STEP 1889 H=1889:00 +CPU= 15.395
  17:53:47 STEP 1890 H=1890:00 +CPU= 24.294
  17:54:03 STEP 1891 H=1891:00 +CPU= 15.686
  17:54:18 STEP 1892 H=1892:00 +CPU= 15.316
  17:54:43 STEP 1893 H=1893:00 +CPU= 24.186
  17:54:59 STEP 1894 H=1894:00 +CPU= 16.184
  17:55:14 STEP 1895 H=1895:00 +CPU= 15.354
  17:55:43 STEP 1896 H=1896:00 +CPU= 28.079
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_78.zip
Uploading the intermediate file: upload_file_78.zip
  17:55:58 STEP 1897 H=1897:00 +CPU= 15.025
  17:56:13 STEP 1898 H=1898:00 +CPU= 15.284
  17:56:37 STEP 1899 H=1899:00 +CPU= 23.505
  17:56:54 STEP 1900 H=1900:00 +CPU= 17.217
  17:57:10 STEP 1901 H=1901:00 +CPU= 15.469
  17:57:34 STEP 1902 H=1902:00 +CPU= 24.485
  17:57:50 STEP 1903 H=1903:00 +CPU= 15.564
  17:58:07 STEP 1904 H=1904:00 +CPU= 16.397
  17:58:30 STEP 1905 H=1905:00 +CPU= 23.370
  17:58:47 STEP 1906 H=1906:00 +CPU= 16.232
  17:59:03 STEP 1907 H=1907:00 +CPU= 16.443
  17:59:33 STEP 1908 H=1908:00 +CPU= 29.454
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001908
  17:59:49 STEP 1909 H=1909:00 +CPU= 16.483
  18:00:05 STEP 1910 H=1910:00 +CPU= 15.871
  18:00:33 STEP 1911 H=1911:00 +CPU= 26.609
  18:00:49 STEP 1912 H=1912:00 +CPU= 16.021
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0295
(argv4) batchid: 983
(argv5) wuid: 12199939
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1b410c501ad82aaef065a533611d1f44 to: /var/lib/boinc/slots/16/oifs_43r3_ps_0295_2014050100_123_983_12199939.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0295_2014050100_123_983_12199939.zip
ic_ancil_file: ic_ancil_12199939
ifsdata_file: ifsdata_12199939
climate_data_file: clim_data_12199939
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3241bf5822a8bc278247a954b047fab8 to: /var/lib/boinc/slots/16/ic_ancil_12199939.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12199939.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bd8aceb907a329b509f0cee8a9b4b6f0 to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12199939.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12199939.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12199939.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12199939.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12199939.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12199939.xml
-- Model is restarting --
Adjusting last_iter, 1912, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12199939.xml
last_cpu_time: 40828
upload_file_number: 79
last_iter: 1895
last_upload: 6825600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083
The child process has been launched with process id: 11746
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:11746:11746] [20230109:184324:1673311404:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:43:54 STEP 1896 H=1896:00 +CPU= 27.239
  18:44:10 STEP 1897 H=1897:00 +CPU= 15.701
  18:44:26 STEP 1898 H=1898:00 +CPU= 15.771
  18:44:51 STEP 1899 H=1899:00 +CPU= 24.366
  18:45:08 STEP 1900 H=1900:00 +CPU= 15.943
  18:45:24 STEP 1901 H=1901:00 +CPU= 15.618
  18:45:49 STEP 1902 H=1902:00 +CPU= 24.411
  18:46:05 STEP 1903 H=1903:00 +CPU= 15.707
  18:46:21 STEP 1904 H=1904:00 +CPU= 16.180
  18:46:46 STEP 1905 H=1905:00 +CPU= 23.990
  18:47:02 STEP 1906 H=1906:00 +CPU= 15.908
  18:47:18 STEP 1907 H=1907:00 +CPU= 15.593
  18:47:46 STEP 1908 H=1908:00 +CPU= 26.999
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001908
  18:48:01 STEP 1909 H=1909:00 +CPU= 15.266
  18:48:17 STEP 1910 H=1910:00 +CPU= 15.820
  18:48:42 STEP 1911 H=1911:00 +CPU= 24.294
  18:48:58 STEP 1912 H=1912:00 +CPU= 15.780
  18:49:14 STEP 1913 H=1913:00 +CPU= 15.398
  18:49:39 STEP 1914 H=1914:00 +CPU= 24.272
  18:49:55 STEP 1915 H=1915:00 +CPU= 15.158
  18:50:11 STEP 1916 H=1916:00 +CPU= 16.102
  18:50:35 STEP 1917 H=1917:00 +CPU= 23.985
  18:50:52 STEP 1918 H=1918:00 +CPU= 15.838
  18:51:08 STEP 1919 H=1919:00 +CPU= 16.017
  18:51:37 STEP 1920 H=1920:00 +CPU= 27.850
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_79.zip
Uploading the intermediate file: upload_file_79.zip
  18:51:53 STEP 1921 H=1921:00 +CPU= 15.609
  18:52:09 STEP 1922 H=1922:00 +CPU= 15.915
  18:52:33 STEP 1923 H=1923:00 +CPU= 23.873
  18:52:50 STEP 1924 H=1924:00 +CPU= 15.821
  18:53:06 STEP 1925 H=1925:00 +CPU= 15.643
  18:53:31 STEP 1926 H=1926:00 +CPU= 24.654
  18:53:47 STEP 1927 H=1927:00 +CPU= 15.473
  18:54:03 STEP 1928 H=1928:00 +CPU= 16.082
  18:54:28 STEP 1929 H=1929:00 +CPU= 23.919
  18:54:44 STEP 1930 H=1930:00 +CPU= 15.753
  18:54:59 STEP 1931 H=1931:00 +CPU= 15.375
  18:55:27 STEP 1932 H=1932:00 +CPU= 26.732
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001932
  18:55:42 STEP 1933 H=1933:00 +CPU= 15.086
  18:55:59 STEP 1934 H=1934:00 +CPU= 16.363
  18:56:23 STEP 1935 H=1935:00 +CPU= 23.728
  18:56:39 STEP 1936 H=1936:00 +CPU= 15.362
  18:56:56 STEP 1937 H=1937:00 +CPU= 16.787
  18:57:21 STEP 1938 H=1938:00 +CPU= 24.198
  18:57:37 STEP 1939 H=1939:00 +CPU= 15.172
  18:57:53 STEP 1940 H=1940:00 +CPU= 15.605
  18:58:17 STEP 1941 H=1941:00 +CPU= 24.025
  18:58:33 STEP 1942 H=1942:00 +CPU= 15.538
  18:58:49 STEP 1943 H=1943:00 +CPU= 15.737
  18:59:18 STEP 1944 H=1944:00 +CPU= 28.330
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_80.zip
Uploading the intermediate file: upload_file_80.zip
  18:59:34 STEP 1945 H=1945:00 +CPU= 15.540
  18:59:50 STEP 1946 H=1946:00 +CPU= 15.686
  19:00:15 STEP 1947 H=1947:00 +CPU= 23.821
  19:00:31 STEP 1948 H=1948:00 +CPU= 15.625
  19:00:46 STEP 1949 H=1949:00 +CPU= 15.271
  19:01:11 STEP 1950 H=1950:00 +CPU= 24.535
  19:01:27 STEP 1951 H=1951:00 +CPU= 15.415
  19:01:43 STEP 1952 H=1952:00 +CPU= 15.831
  19:02:08 STEP 1953 H=1953:00 +CPU= 23.888
  19:02:24 STEP 1954 H=1954:00 +CPU= 15.990
  19:02:40 STEP 1955 H=1955:00 +CPU= 15.404
  19:03:07 STEP 1956 H=1956:00 +CPU= 26.717
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001956
  19:03:23 STEP 1957 H=1957:00 +CPU= 15.171
  19:03:39 STEP 1958 H=1958:00 +CPU= 15.837
  19:04:03 STEP 1959 H=1959:00 +CPU= 23.873
  19:04:20 STEP 1960 H=1960:00 +CPU= 15.690
  19:04:35 STEP 1961 H=1961:00 +CPU= 15.376
  19:05:00 STEP 1962 H=1962:00 +CPU= 24.142
  19:05:16 STEP 1963 H=1963:00 +CPU= 15.479
  19:05:32 STEP 1964 H=1964:00 +CPU= 15.896
  19:05:56 STEP 1965 H=1965:00 +CPU= 23.767
  19:06:12 STEP 1966 H=1966:00 +CPU= 15.393
  19:06:28 STEP 1967 H=1967:00 +CPU= 15.935
  19:06:57 STEP 1968 H=1968:00 +CPU= 27.795
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_81.zip
Uploading the intermediate file: upload_file_81.zip
  19:07:13 STEP 1969 H=1969:00 +CPU= 15.716
  19:07:29 STEP 1970 H=1970:00 +CPU= 15.556
  19:07:53 STEP 1971 H=1971:00 +CPU= 23.524
  19:08:08 STEP 1972 H=1972:00 +CPU= 15.512
  19:08:24 STEP 1973 H=1973:00 +CPU= 15.275
  19:08:47 STEP 1974 H=1974:00 +CPU= 23.029
  19:09:02 STEP 1975 H=1975:00 +CPU= 14.499
  19:09:17 STEP 1976 H=1976:00 +CPU= 15.455
  19:09:41 STEP 1977 H=1977:00 +CPU= 23.533
  19:09:57 STEP 1978 H=1978:00 +CPU= 15.575
  19:10:13 STEP 1979 H=1979:00 +CPU= 15.757
  19:10:40 STEP 1980 H=1980:00 +CPU= 26.590
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001980
  19:10:56 STEP 1981 H=1981:00 +CPU= 15.387
  19:11:12 STEP 1982 H=1982:00 +CPU= 15.774
  19:11:37 STEP 1983 H=1983:00 +CPU= 23.957
  19:11:53 STEP 1984 H=1984:00 +CPU= 16.075
  19:12:10 STEP 1985 H=1985:00 +CPU= 15.836
  19:12:36 STEP 1986 H=1986:00 +CPU= 25.561
  19:12:51 STEP 1987 H=1987:00 +CPU= 15.362
  19:13:08 STEP 1988 H=1988:00 +CPU= 16.044
  19:13:33 STEP 1989 H=1989:00 +CPU= 24.427
  19:13:50 STEP 1990 H=1990:00 +CPU= 16.362
  19:14:06 STEP 1991 H=1991:00 +CPU= 16.213
  19:14:36 STEP 1992 H=1992:00 +CPU= 28.680
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_82.zip
Uploading the intermediate file: upload_file_82.zip
  19:14:52 STEP 1993 H=1993:00 +CPU= 15.680
  19:15:08 STEP 1994 H=1994:00 +CPU= 15.917
  19:15:33 STEP 1995 H=1995:00 +CPU= 24.237
  19:15:50 STEP 1996 H=1996:00 +CPU= 16.193
  19:16:06 STEP 1997 H=1997:00 +CPU= 15.407
  19:16:31 STEP 1998 H=1998:00 +CPU= 24.893
  19:16:47 STEP 1999 H=1999:00 +CPU= 15.692
  19:17:04 STEP 2000 H=2000:00 +CPU= 16.237
  19:17:29 STEP 2001 H=2001:00 +CPU= 24.391
  19:17:45 STEP 2002 H=2002:00 +CPU= 16.061
  19:18:01 STEP 2003 H=2003:00 +CPU= 15.506
  19:18:29 STEP 2004 H=2004:00 +CPU= 26.942
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002004
  19:18:44 STEP 2005 H=2005:00 +CPU= 15.450
  19:19:01 STEP 2006 H=2006:00 +CPU= 15.872
  19:19:26 STEP 2007 H=2007:00 +CPU= 24.294
  19:19:42 STEP 2008 H=2008:00 +CPU= 15.958
  19:19:58 STEP 2009 H=2009:00 +CPU= 15.743
  19:20:24 STEP 2010 H=2010:00 +CPU= 25.046
  19:20:40 STEP 2011 H=2011:00 +CPU= 15.612
  19:20:56 STEP 2012 H=2012:00 +CPU= 15.927
  19:21:22 STEP 2013 H=2013:00 +CPU= 24.820
  19:21:38 STEP 2014 H=2014:00 +CPU= 16.110
  19:21:55 STEP 2015 H=2015:00 +CPU= 16.114
  19:22:24 STEP 2016 H=2016:00 +CPU= 29.008
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_83.zip
Uploading the intermediate file: upload_file_83.zip
  19:22:40 STEP 2017 H=2017:00 +CPU= 15.625
  19:22:57 STEP 2018 H=2018:00 +CPU= 16.339
  19:23:22 STEP 2019 H=2019:00 +CPU= 24.373
  19:23:38 STEP 2020 H=2020:00 +CPU= 15.947
  19:23:55 STEP 2021 H=2021:00 +CPU= 15.883
  19:24:20 STEP 2022 H=2022:00 +CPU= 24.406
  19:24:35 STEP 2023 H=2023:00 +CPU= 14.737
  19:24:50 STEP 2024 H=2024:00 +CPU= 15.103
  19:25:13 STEP 2025 H=2025:00 +CPU= 22.747
  19:25:29 STEP 2026 H=2026:00 +CPU= 15.262
  19:25:44 STEP 2027 H=2027:00 +CPU= 14.947
  19:26:12 STEP 2028 H=2028:00 +CPU= 27.403
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002028
  19:26:28 STEP 2029 H=2029:00 +CPU= 15.814
  19:26:45 STEP 2030 H=2030:00 +CPU= 16.249
  19:27:11 STEP 2031 H=2031:00 +CPU= 24.770
  19:27:27 STEP 2032 H=2032:00 +CPU= 16.208
  19:27:44 STEP 2033 H=2033:00 +CPU= 15.665
  19:28:09 STEP 2034 H=2034:00 +CPU= 25.047
  19:28:25 STEP 2035 H=2035:00 +CPU= 15.551
  19:28:42 STEP 2036 H=2036:00 +CPU= 16.044
  19:29:07 STEP 2037 H=2037:00 +CPU= 24.471
  19:29:24 STEP 2038 H=2038:00 +CPU= 16.089
  19:29:40 STEP 2039 H=2039:00 +CPU= 16.075
  19:30:10 STEP 2040 H=2040:00 +CPU= 28.669
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_84.zip
Uploading the intermediate file: upload_file_84.zip
  19:30:26 STEP 2041 H=2041:00 +CPU= 15.589
  19:30:42 STEP 2042 H=2042:00 +CPU= 16.056
  19:31:07 STEP 2043 H=2043:00 +CPU= 24.297
  19:31:24 STEP 2044 H=2044:00 +CPU= 16.250
  19:31:40 STEP 2045 H=2045:00 +CPU= 15.560
  19:32:06 STEP 2046 H=2046:00 +CPU= 24.876
  19:32:22 STEP 2047 H=2047:00 +CPU= 15.713
  19:32:39 STEP 2048 H=2048:00 +CPU= 16.340
  19:33:04 STEP 2049 H=2049:00 +CPU= 24.611
  19:33:21 STEP 2050 H=2050:00 +CPU= 16.131
  19:33:37 STEP 2051 H=2051:00 +CPU= 15.832
  19:34:05 STEP 2052 H=2052:00 +CPU= 27.486
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002052
  19:34:22 STEP 2053 H=2053:00 +CPU= 15.636
  19:34:38 STEP 2054 H=2054:00 +CPU= 16.200
  19:35:04 STEP 2055 H=2055:00 +CPU= 24.740
  19:35:21 STEP 2056 H=2056:00 +CPU= 16.355
  19:35:37 STEP 2057 H=2057:00 +CPU= 15.807
  19:36:03 STEP 2058 H=2058:00 +CPU= 24.854
  19:36:19 STEP 2059 H=2059:00 +CPU= 15.532
  19:36:36 STEP 2060 H=2060:00 +CPU= 16.277
  19:37:01 STEP 2061 H=2061:00 +CPU= 24.559
  19:37:18 STEP 2062 H=2062:00 +CPU= 16.275
  19:37:34 STEP 2063 H=2063:00 +CPU= 16.098
  19:38:05 STEP 2064 H=2064:00 +CPU= 29.069
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_85.zip
Uploading the intermediate file: upload_file_85.zip
  19:38:21 STEP 2065 H=2065:00 +CPU= 15.762
  19:38:38 STEP 2066 H=2066:00 +CPU= 16.254
  19:39:03 STEP 2067 H=2067:00 +CPU= 24.904
  19:39:18 STEP 2068 H=2068:00 +CPU= 14.935
  19:39:34 STEP 2069 H=2069:00 +CPU= 14.959
  19:39:57 STEP 2070 H=2070:00 +CPU= 23.311
  19:40:12 STEP 2071 H=2071:00 +CPU= 14.692
  19:40:27 STEP 2072 H=2072:00 +CPU= 15.247
  19:40:53 STEP 2073 H=2073:00 +CPU= 25.139
  19:41:10 STEP 2074 H=2074:00 +CPU= 16.595
  19:41:25 STEP 2075 H=2075:00 +CPU= 14.823
  19:41:51 STEP 2076 H=2076:00 +CPU= 25.640
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002076
  19:42:06 STEP 2077 H=2077:00 +CPU= 14.814
  19:42:21 STEP 2078 H=2078:00 +CPU= 14.909
  19:42:44 STEP 2079 H=2079:00 +CPU= 22.540
  19:42:59 STEP 2080 H=2080:00 +CPU= 14.934
  19:43:13 STEP 2081 H=2081:00 +CPU= 14.265
  19:43:36 STEP 2082 H=2082:00 +CPU= 22.689
  19:43:50 STEP 2083 H=2083:00 +CPU= 14.234
  19:44:05 STEP 2084 H=2084:00 +CPU= 15.325
  19:44:28 STEP 2085 H=2085:00 +CPU= 22.556
  19:44:43 STEP 2086 H=2086:00 +CPU= 14.586
  19:44:57 STEP 2087 H=2087:00 +CPU= 14.058
  19:45:24 STEP 2088 H=2088:00 +CPU= 26.734
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_86.zip
Uploading the intermediate file: upload_file_86.zip
  19:45:39 STEP 2089 H=2089:00 +CPU= 14.838
  19:45:54 STEP 2090 H=2090:00 +CPU= 14.732
  19:46:16 STEP 2091 H=2091:00 +CPU= 22.264
  19:46:30 STEP 2092 H=2092:00 +CPU= 14.281
  19:46:45 STEP 2093 H=2093:00 +CPU= 14.314
  19:47:09 STEP 2094 H=2094:00 +CPU= 23.576
  19:47:23 STEP 2095 H=2095:00 +CPU= 14.505
  19:47:38 STEP 2096 H=2096:00 +CPU= 14.987
  19:48:01 STEP 2097 H=2097:00 +CPU= 22.679
  19:48:16 STEP 2098 H=2098:00 +CPU= 14.569
  19:48:30 STEP 2099 H=2099:00 +CPU= 14.525
  19:48:55 STEP 2100 H=2100:00 +CPU= 24.415
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002100
  19:49:09 STEP 2101 H=2101:00 +CPU= 14.196
  19:49:24 STEP 2102 H=2102:00 +CPU= 14.946
  19:49:47 STEP 2103 H=2103:00 +CPU= 22.796
  19:50:02 STEP 2104 H=2104:00 +CPU= 14.788
  19:50:16 STEP 2105 H=2105:00 +CPU= 14.153
  19:50:39 STEP 2106 H=2106:00 +CPU= 22.852
  19:50:53 STEP 2107 H=2107:00 +CPU= 13.902
  19:51:08 STEP 2108 H=2108:00 +CPU= 14.679
  19:51:32 STEP 2109 H=2109:00 +CPU= 23.692
  19:51:47 STEP 2110 H=2110:00 +CPU= 15.350
  19:52:01 STEP 2111 H=2111:00 +CPU= 14.064
  19:52:28 STEP 2112 H=2112:00 +CPU= 26.719
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_87.zip
Uploading the intermediate file: upload_file_87.zip
  19:52:42 STEP 2113 H=2113:00 +CPU= 13.526
  19:52:57 STEP 2114 H=2114:00 +CPU= 15.155
  19:53:20 STEP 2115 H=2115:00 +CPU= 22.966
  19:53:34 STEP 2116 H=2116:00 +CPU= 13.848
  19:53:49 STEP 2117 H=2117:00 +CPU= 14.812
  19:54:11 STEP 2118 H=2118:00 +CPU= 22.513
  19:54:25 STEP 2119 H=2119:00 +CPU= 13.977
  19:54:41 STEP 2120 H=2120:00 +CPU= 14.957
  19:55:01 STEP 2121 H=2121:00 +CPU= 20.173
  19:55:14 STEP 2122 H=2122:00 +CPU= 13.452
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0295
(argv4) batchid: 983
(argv5) wuid: 12199939
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/16
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1b410c501ad82aaef065a533611d1f44 to: /var/lib/boinc/slots/16/oifs_43r3_ps_0295_2014050100_123_983_12199939.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/16/oifs_43r3_ps_0295_2014050100_123_983_12199939.zip
ic_ancil_file: ic_ancil_12199939
ifsdata_file: ifsdata_12199939
climate_data_file: clim_data_12199939
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3241bf5822a8bc278247a954b047fab8 to: /var/lib/boinc/slots/16/ic_ancil_12199939.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/16/ic_ancil_12199939.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bd8aceb907a329b509f0cee8a9b4b6f0 to: /var/lib/boinc/slots/16/ifsdata/ifsdata_12199939.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/16/ifsdata/ifsdata_12199939.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/16/159l_2/clim_data_12199939.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/16/159l_2/clim_data_12199939.zip
Checking for progress XML file: /var/lib/boinc/slots/16/progress_file_12199939.xml
Opened progress file ok : /var/lib/boinc/slots/16/progress_file_12199939.xml
-- Model is restarting --
Adjusting last_iter, 2123, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/16/progress_file_12199939.xml
last_cpu_time: 45052
upload_file_number: 88
last_iter: 2111
last_upload: 7603200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083
The child process has been launched with process id: 39647
Executing the command: /var/lib/boinc/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Eggshelsior-Manjaro:1:1:39647:39647] [20230109:201809:1673317089:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:18:37 STEP 2112 H=2112:00 +CPU= 27.083
  20:18:53 STEP 2113 H=2113:00 +CPU= 16.030
  20:19:09 STEP 2114 H=2114:00 +CPU= 15.486
  20:19:34 STEP 2115 H=2115:00 +CPU= 25.060
  20:19:50 STEP 2116 H=2116:00 +CPU= 16.120
  20:20:06 STEP 2117 H=2117:00 +CPU= 15.742
  20:20:34 STEP 2118 H=2118:00 +CPU= 27.796
  20:20:49 STEP 2119 H=2119:00 +CPU= 14.826
  20:21:04 STEP 2120 H=2120:00 +CPU= 15.016
  20:21:27 STEP 2121 H=2121:00 +CPU= 21.993
  20:21:42 STEP 2122 H=2122:00 +CPU= 15.420
  20:21:56 STEP 2123 H=2123:00 +CPU= 13.976
  20:22:23 STEP 2124 H=2124:00 +CPU= 26.487
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002124
  20:22:37 STEP 2125 H=2125:00 +CPU= 13.699
  20:22:52 STEP 2126 H=2126:00 +CPU= 14.719
  20:23:13 STEP 2127 H=2127:00 +CPU= 21.336
  20:23:27 STEP 2128 H=2128:00 +CPU= 13.837
  20:23:41 STEP 2129 H=2129:00 +CPU= 13.629
  20:24:04 STEP 2130 H=2130:00 +CPU= 22.697
  20:24:18 STEP 2131 H=2131:00 +CPU= 14.792
  20:24:33 STEP 2132 H=2132:00 +CPU= 14.965
  20:24:57 STEP 2133 H=2133:00 +CPU= 23.813
  20:25:12 STEP 2134 H=2134:00 +CPU= 13.994
  20:25:25 STEP 2135 H=2135:00 +CPU= 13.927
  20:25:52 STEP 2136 H=2136:00 +CPU= 26.697
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12199939/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0295_2014050100_123_983_12199939_0_r1359440083_88.zip
Uploading the intermediate file: upload_file_88.zip
  20:26:07 STEP 2137 H=2137:00 +CPU= 14.118
  20:26:21 STEP 2138 H=2138:00 +CPU= 14.496
  20:26:43 STEP 2139 H=2139:00 +CPU= 21.894
  20:26:58 STEP 2140 H=2140:00 +CPU= 14.444
  20:27:12 STEP 2141 H=2141:00 +CPU= 13.854
  20:27:34 STEP 2142 H=2142:00 +CPU= 22.320
  20:27:49 STEP 2143 H=2143:00 +CPU= 14.116
  20:28:03 STEP 2144 H=2144:00 +CPU= 14.392
  20:28:26 STEP 2145 H=2145:00 +CPU= 22.365
  20:28:40 STEP 2146 H=2146:00 +CPU= 14.333
  20:28:54 STEP 2147 H=2147:00 +CPU= 13.963
  20:29:19 STEP 2148 H=2148:00 +CPU= 24.904
Moving to projects directory: /var/lib/boinc/slots/16/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/16/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc/slots/16/ICMUAhq0f+002148
  20:29:34 STEP 2149 H=2149:00 +CPU= 14.611
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:14:10 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 6,566,400 39,176 0.0060
09 May 2023 16:14:05 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 5,702,400 34,386 0.0060
09 May 2023 16:14:02 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 4,838,400 29,817 0.0062
09 May 2023 16:13:45 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 3,024,000 18,716 0.0062
09 May 2023 16:13:42 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 2,160,000 13,073 0.0061
09 May 2023 16:13:31 1502085 22286148 oifs_43r3_ps_0295_2014050100_123_983_12199939_0 864,000 4,972 0.0058


©2024 climateprediction.net