climateprediction.net home page
Task 22282166

Task 22282166

Name oifs_43r3_ps_0561_2010050100_123_979_12196205_0
Workunit 12196205
Created 23 Dec 2022, 23:59:34 UTC
Sent 24 Dec 2022, 0:11:56 UTC
Report deadline 23 Jan 2023, 0:11:56 UTC
Received 11 Jan 2023, 20:07:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537350
Run time 19 hours 30 min 56 sec
CPU time 19 hours 27 min 24 sec
Validate state Invalid
Credit 10,886.40
Device peak FLOPS 5.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.39 MB
Peak swap size 4,978.08 MB
Peak disk usage 1,959.77 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
96205/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_76.zip
Uploading the intermediate file: upload_file_76.zip
  15:30:31 STEP 1849 H=1849:00 +CPU= 25.195
  15:30:55 STEP 1850 H=1850:00 +CPU= 23.567
  15:31:31 STEP 1851 H=1851:00 +CPU= 36.167
  15:31:57 STEP 1852 H=1852:00 +CPU= 25.774
  15:32:22 STEP 1853 H=1853:00 +CPU= 25.555
  15:33:02 STEP 1854 H=1854:00 +CPU= 40.128
  15:33:28 STEP 1855 H=1855:00 +CPU= 25.984
  15:33:55 STEP 1856 H=1856:00 +CPU= 26.543
  15:34:34 STEP 1857 H=1857:00 +CPU= 38.923
  15:34:59 STEP 1858 H=1858:00 +CPU= 25.254
  15:35:24 STEP 1859 H=1859:00 +CPU= 24.553
  15:36:06 STEP 1860 H=1860:00 +CPU= 42.393
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001860
  15:36:31 STEP 1861 H=1861:00 +CPU= 25.053
  15:36:57 STEP 1862 H=1862:00 +CPU= 26.078
  15:37:36 STEP 1863 H=1863:00 +CPU= 39.279
  15:38:02 STEP 1864 H=1864:00 +CPU= 25.961
  15:38:28 STEP 1865 H=1865:00 +CPU= 25.613
  15:39:06 STEP 1866 H=1866:00 +CPU= 38.251
  15:39:30 STEP 1867 H=1867:00 +CPU= 23.865
  15:39:55 STEP 1868 H=1868:00 +CPU= 24.746
  15:40:36 STEP 1869 H=1869:00 +CPU= 40.730
  15:41:02 STEP 1870 H=1870:00 +CPU= 25.861
  15:41:28 STEP 1871 H=1871:00 +CPU= 26.206
  15:42:14 STEP 1872 H=1872:00 +CPU= 46.298
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_77.zip
Uploading the intermediate file: upload_file_77.zip
  15:42:39 STEP 1873 H=1873:00 +CPU= 25.056
  15:43:05 STEP 1874 H=1874:00 +CPU= 25.605
  15:43:44 STEP 1875 H=1875:00 +CPU= 39.446
  15:44:10 STEP 1876 H=1876:00 +CPU= 25.844
  15:44:36 STEP 1877 H=1877:00 +CPU= 25.235
  15:45:15 STEP 1878 H=1878:00 +CPU= 39.842
  15:45:41 STEP 1879 H=1879:00 +CPU= 25.408
  15:46:07 STEP 1880 H=1880:00 +CPU= 25.994
  15:46:45 STEP 1881 H=1881:00 +CPU= 38.252
  15:47:10 STEP 1882 H=1882:00 +CPU= 24.560
  15:47:34 STEP 1883 H=1883:00 +CPU= 23.881
  15:48:15 STEP 1884 H=1884:00 +CPU= 41.622
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001884
  15:48:41 STEP 1885 H=1885:00 +CPU= 25.532
  15:49:06 STEP 1886 H=1886:00 +CPU= 25.397
  15:49:46 STEP 1887 H=1887:00 +CPU= 40.092
  15:50:12 STEP 1888 H=1888:00 +CPU= 26.274
  15:50:35 STEP 1889 H=1889:00 +CPU= 22.527
  15:51:15 STEP 1890 H=1890:00 +CPU= 39.518
  15:51:40 STEP 1891 H=1891:00 +CPU= 25.037
  15:52:06 STEP 1892 H=1892:00 +CPU= 26.115
  15:52:44 STEP 1893 H=1893:00 +CPU= 38.451
  15:53:08 STEP 1894 H=1894:00 +CPU= 24.018
  15:53:34 STEP 1895 H=1895:00 +CPU= 26.162
  15:54:21 STEP 1896 H=1896:00 +CPU= 46.159
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_78.zip
Uploading the intermediate file: upload_file_78.zip
  15:54:46 STEP 1897 H=1897:00 +CPU= 25.730
  15:55:12 STEP 1898 H=1898:00 +CPU= 25.738
  15:55:52 STEP 1899 H=1899:00 +CPU= 39.508
  15:56:17 STEP 1900 H=1900:00 +CPU= 25.545
  15:56:41 STEP 1901 H=1901:00 +CPU= 24.075
  15:57:22 STEP 1902 H=1902:00 +CPU= 40.300
  15:57:48 STEP 1903 H=1903:00 +CPU= 25.925
  15:58:14 STEP 1904 H=1904:00 +CPU= 26.163
  15:58:52 STEP 1905 H=1905:00 +CPU= 38.404
  15:59:18 STEP 1906 H=1906:00 +CPU= 25.419
  15:59:42 STEP 1907 H=1907:00 +CPU= 24.463
  16:00:22 STEP 1908 H=1908:00 +CPU= 40.024
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001908
  16:00:45 STEP 1909 H=1909:00 +CPU= 22.596
  16:01:06 STEP 1910 H=1910:00 +CPU= 21.465
  16:01:39 STEP 1911 H=1911:00 +CPU= 33.001
  16:02:03 STEP 1912 H=1912:00 +CPU= 24.028
  16:02:29 STEP 1913 H=1913:00 +CPU= 25.741
  16:03:09 STEP 1914 H=1914:00 +CPU= 40.107
  16:03:35 STEP 1915 H=1915:00 +CPU= 25.361
  16:04:00 STEP 1916 H=1916:00 +CPU= 24.916
  16:04:39 STEP 1917 H=1917:00 +CPU= 39.466
  16:05:05 STEP 1918 H=1918:00 +CPU= 26.112
  16:05:32 STEP 1919 H=1919:00 +CPU= 27.044
  16:06:20 STEP 1920 H=1920:00 +CPU= 47.585
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_79.zip
Uploading the intermediate file: upload_file_79.zip
  16:06:46 STEP 1921 H=1921:00 +CPU= 25.603
Uploading trickle at timestep: 6912000
  16:07:12 STEP 1922 H=1922:00 +CPU= 26.104
  16:07:51 STEP 1923 H=1923:00 +CPU= 39.333
  16:08:17 STEP 1924 H=1924:00 +CPU= 26.408
  16:08:43 STEP 1925 H=1925:00 +CPU= 25.198
  16:09:23 STEP 1926 H=1926:00 +CPU= 40.501
  16:09:49 STEP 1927 H=1927:00 +CPU= 25.396
  16:10:14 STEP 1928 H=1928:00 +CPU= 25.871
  16:10:53 STEP 1929 H=1929:00 +CPU= 38.464
  16:11:19 STEP 1930 H=1930:00 +CPU= 25.993
  16:11:43 STEP 1931 H=1931:00 +CPU= 24.564
  16:12:27 STEP 1932 H=1932:00 +CPU= 43.520
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001932
  16:12:52 STEP 1933 H=1933:00 +CPU= 25.044
  16:13:17 STEP 1934 H=1934:00 +CPU= 24.715
  16:13:54 STEP 1935 H=1935:00 +CPU= 36.813
  16:14:16 STEP 1936 H=1936:00 +CPU= 21.941
  16:14:40 STEP 1937 H=1937:00 +CPU= 24.564
  16:15:14 STEP 1938 H=1938:00 +CPU= 33.818
  16:15:39 STEP 1939 H=1939:00 +CPU= 24.562
  16:16:05 STEP 1940 H=1940:00 +CPU= 26.287
  16:16:45 STEP 1941 H=1941:00 +CPU= 39.853
  16:17:09 STEP 1942 H=1942:00 +CPU= 24.213
  16:17:35 STEP 1943 H=1943:00 +CPU= 26.219
  16:18:22 STEP 1944 H=1944:00 +CPU= 46.382
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_80.zip
Uploading the intermediate file: upload_file_80.zip
  16:18:47 STEP 1945 H=1945:00 +CPU= 24.887
  16:19:13 STEP 1946 H=1946:00 +CPU= 26.335
  16:19:52 STEP 1947 H=1947:00 +CPU= 39.102
  16:20:18 STEP 1948 H=1948:00 +CPU= 25.911
  16:20:42 STEP 1949 H=1949:00 +CPU= 24.059
  16:21:21 STEP 1950 H=1950:00 +CPU= 38.985
  16:21:46 STEP 1951 H=1951:00 +CPU= 25.006
  16:22:13 STEP 1952 H=1952:00 +CPU= 26.376
  16:22:51 STEP 1953 H=1953:00 +CPU= 38.791
  16:23:17 STEP 1954 H=1954:00 +CPU= 25.666
  16:23:42 STEP 1955 H=1955:00 +CPU= 25.034
  16:24:26 STEP 1956 H=1956:00 +CPU= 44.210
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001956
  16:24:52 STEP 1957 H=1957:00 +CPU= 25.338
  16:25:18 STEP 1958 H=1958:00 +CPU= 25.862
  16:25:56 STEP 1959 H=1959:00 +CPU= 38.872
  16:26:23 STEP 1960 H=1960:00 +CPU= 26.500
  16:26:46 STEP 1961 H=1961:00 +CPU= 23.118
  16:27:27 STEP 1962 H=1962:00 +CPU= 40.790
  16:27:51 STEP 1963 H=1963:00 +CPU= 24.152
  16:28:16 STEP 1964 H=1964:00 +CPU= 24.514
  16:28:55 STEP 1965 H=1965:00 +CPU= 39.618
  16:29:22 STEP 1966 H=1966:00 +CPU= 26.087
  16:29:46 STEP 1967 H=1967:00 +CPU= 24.635
  16:30:31 STEP 1968 H=1968:00 +CPU= 44.418
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_81.zip
Uploading the intermediate file: upload_file_81.zip
  16:30:56 STEP 1969 H=1969:00 +CPU= 25.289
  16:31:20 STEP 1970 H=1970:00 +CPU= 24.142
  16:31:54 STEP 1971 H=1971:00 +CPU= 33.261
  16:32:18 STEP 1972 H=1972:00 +CPU= 24.031
  16:32:41 STEP 1973 H=1973:00 +CPU= 23.405
  16:33:18 STEP 1974 H=1974:00 +CPU= 37.242
  16:33:44 STEP 1975 H=1975:00 +CPU= 25.425
  16:34:10 STEP 1976 H=1976:00 +CPU= 26.228
  16:34:50 STEP 1977 H=1977:00 +CPU= 39.453
  16:35:15 STEP 1978 H=1978:00 +CPU= 25.169
  16:35:38 STEP 1979 H=1979:00 +CPU= 23.345
  16:36:19 STEP 1980 H=1980:00 +CPU= 40.718
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001980
  16:36:44 STEP 1981 H=1981:00 +CPU= 25.359
  16:37:10 STEP 1982 H=1982:00 +CPU= 26.208
  16:37:50 STEP 1983 H=1983:00 +CPU= 39.343
  16:38:16 STEP 1984 H=1984:00 +CPU= 26.291
  16:38:42 STEP 1985 H=1985:00 +CPU= 25.463
  16:39:22 STEP 1986 H=1986:00 +CPU= 40.047
  16:39:46 STEP 1987 H=1987:00 +CPU= 24.294
  16:40:10 STEP 1988 H=1988:00 +CPU= 24.146
  16:40:47 STEP 1989 H=1989:00 +CPU= 37.259
  16:41:14 STEP 1990 H=1990:00 +CPU= 26.632
  16:41:41 STEP 1991 H=1991:00 +CPU= 26.503
  16:42:25 STEP 1992 H=1992:00 +CPU= 44.695
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_82.zip
Uploading the intermediate file: upload_file_82.zip
  16:42:50 STEP 1993 H=1993:00 +CPU= 24.368
  16:43:16 STEP 1994 H=1994:00 +CPU= 25.903
  16:43:52 STEP 1995 H=1995:00 +CPU= 36.522
  16:44:16 STEP 1996 H=1996:00 +CPU= 23.734
  16:44:41 STEP 1997 H=1997:00 +CPU= 24.920
  16:45:19 STEP 1998 H=1998:00 +CPU= 38.088
  16:45:44 STEP 1999 H=1999:00 +CPU= 24.618
  16:46:08 STEP 2000 H=2000:00 +CPU= 24.483
  16:46:45 STEP 2001 H=2001:00 +CPU= 37.445
  16:47:11 STEP 2002 H=2002:00 +CPU= 25.462
  16:47:36 STEP 2003 H=2003:00 +CPU= 24.772
  16:48:17 STEP 2004 H=2004:00 +CPU= 40.865
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002004
  16:48:40 STEP 2005 H=2005:00 +CPU= 23.609
  16:49:01 STEP 2006 H=2006:00 +CPU= 20.365
  16:49:39 STEP 2007 H=2007:00 +CPU= 38.749
  16:50:05 STEP 2008 H=2008:00 +CPU= 25.715
  16:50:30 STEP 2009 H=2009:00 +CPU= 25.186
  16:51:09 STEP 2010 H=2010:00 +CPU= 39.043
  16:51:36 STEP 2011 H=2011:00 +CPU= 26.230
  16:52:01 STEP 2012 H=2012:00 +CPU= 25.135
  16:52:40 STEP 2013 H=2013:00 +CPU= 38.758
  16:53:04 STEP 2014 H=2014:00 +CPU= 24.826
  16:53:31 STEP 2015 H=2015:00 +CPU= 26.170
  16:54:16 STEP 2016 H=2016:00 +CPU= 45.045
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_83.zip
Uploading the intermediate file: upload_file_83.zip
  16:54:42 STEP 2017 H=2017:00 +CPU= 25.995
  16:55:05 STEP 2018 H=2018:00 +CPU= 23.718
  16:55:43 STEP 2019 H=2019:00 +CPU= 37.654
  16:56:09 STEP 2020 H=2020:00 +CPU= 25.298
  16:56:32 STEP 2021 H=2021:00 +CPU= 23.283
  16:57:12 STEP 2022 H=2022:00 +CPU= 40.374
  16:57:38 STEP 2023 H=2023:00 +CPU= 25.840
  16:58:04 STEP 2024 H=2024:00 +CPU= 25.657
  16:58:39 STEP 2025 H=2025:00 +CPU= 35.482
  16:59:04 STEP 2026 H=2026:00 +CPU= 25.006
  16:59:30 STEP 2027 H=2027:00 +CPU= 25.146
  17:00:14 STEP 2028 H=2028:00 +CPU= 44.264
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002028
  17:00:40 STEP 2029 H=2029:00 +CPU= 26.332
  17:01:05 STEP 2030 H=2030:00 +CPU= 24.407
  17:01:40 STEP 2031 H=2031:00 +CPU= 35.389
  17:02:07 STEP 2032 H=2032:00 +CPU= 27.132
  17:02:33 STEP 2033 H=2033:00 +CPU= 25.819
  17:03:14 STEP 2034 H=2034:00 +CPU= 40.596
  17:03:39 STEP 2035 H=2035:00 +CPU= 25.401
  17:04:05 STEP 2036 H=2036:00 +CPU= 25.881
  17:04:41 STEP 2037 H=2037:00 +CPU= 36.058
  17:05:07 STEP 2038 H=2038:00 +CPU= 25.999
  17:05:32 STEP 2039 H=2039:00 +CPU= 24.834
  17:06:19 STEP 2040 H=2040:00 +CPU= 46.450
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_84.zip
Uploading the intermediate file: upload_file_84.zip
  17:06:43 STEP 2041 H=2041:00 +CPU= 24.094
  17:07:08 STEP 2042 H=2042:00 +CPU= 25.307
  17:07:46 STEP 2043 H=2043:00 +CPU= 38.066
  17:08:12 STEP 2044 H=2044:00 +CPU= 25.390
  17:08:36 STEP 2045 H=2045:00 +CPU= 24.852
  17:09:16 STEP 2046 H=2046:00 +CPU= 39.323
  17:09:41 STEP 2047 H=2047:00 +CPU= 25.651
  17:10:07 STEP 2048 H=2048:00 +CPU= 25.746
  17:10:45 STEP 2049 H=2049:00 +CPU= 37.763
  17:11:10 STEP 2050 H=2050:00 +CPU= 24.530
  17:11:35 STEP 2051 H=2051:00 +CPU= 25.504
  17:12:19 STEP 2052 H=2052:00 +CPU= 43.638
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002052
  17:12:44 STEP 2053 H=2053:00 +CPU= 25.451
  17:13:10 STEP 2054 H=2054:00 +CPU= 25.449
  17:13:47 STEP 2055 H=2055:00 +CPU= 37.538
  17:14:10 STEP 2056 H=2056:00 +CPU= 22.699
  17:14:36 STEP 2057 H=2057:00 +CPU= 25.522
  17:15:16 STEP 2058 H=2058:00 +CPU= 40.162
  17:15:36 STEP 2059 H=2059:00 +CPU= 19.984
  17:16:02 STEP 2060 H=2060:00 +CPU= 25.748
  17:16:40 STEP 2061 H=2061:00 +CPU= 38.124
  17:17:04 STEP 2062 H=2062:00 +CPU= 24.563
  17:17:28 STEP 2063 H=2063:00 +CPU= 24.207
  17:18:14 STEP 2064 H=2064:00 +CPU= 45.783
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_85.zip
Uploading the intermediate file: upload_file_85.zip
  17:18:40 STEP 2065 H=2065:00 +CPU= 25.230
  17:19:05 STEP 2066 H=2066:00 +CPU= 25.885
  17:19:45 STEP 2067 H=2067:00 +CPU= 39.671
  17:20:12 STEP 2068 H=2068:00 +CPU= 26.589
  17:20:34 STEP 2069 H=2069:00 +CPU= 22.703
  17:21:15 STEP 2070 H=2070:00 +CPU= 40.069
  17:21:40 STEP 2071 H=2071:00 +CPU= 25.725
  17:22:06 STEP 2072 H=2072:00 +CPU= 26.123
  17:22:47 STEP 2073 H=2073:00 +CPU= 40.184
  17:23:13 STEP 2074 H=2074:00 +CPU= 26.322
  17:23:38 STEP 2075 H=2075:00 +CPU= 24.836
  17:24:22 STEP 2076 H=2076:00 +CPU= 43.876
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002076
  17:24:48 STEP 2077 H=2077:00 +CPU= 26.003
  17:25:14 STEP 2078 H=2078:00 +CPU= 25.842
  17:25:53 STEP 2079 H=2079:00 +CPU= 39.908
  17:26:20 STEP 2080 H=2080:00 +CPU= 26.553
  17:26:43 STEP 2081 H=2081:00 +CPU= 22.876
  17:27:23 STEP 2082 H=2082:00 +CPU= 40.096
  17:27:49 STEP 2083 H=2083:00 +CPU= 25.849
  17:28:14 STEP 2084 H=2084:00 +CPU= 25.257
  17:28:54 STEP 2085 H=2085:00 +CPU= 39.257
  17:29:20 STEP 2086 H=2086:00 +CPU= 26.109
  17:29:46 STEP 2087 H=2087:00 +CPU= 26.187
  17:30:30 STEP 2088 H=2088:00 +CPU= 43.891
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_86.zip
Uploading the intermediate file: upload_file_86.zip
  17:30:55 STEP 2089 H=2089:00 +CPU= 24.850
  17:31:20 STEP 2090 H=2090:00 +CPU= 25.148
  17:31:55 STEP 2091 H=2091:00 +CPU= 35.200
  17:32:20 STEP 2092 H=2092:00 +CPU= 24.644
  17:32:45 STEP 2093 H=2093:00 +CPU= 25.491
  17:33:23 STEP 2094 H=2094:00 +CPU= 37.798
  17:33:49 STEP 2095 H=2095:00 +CPU= 26.328
  17:34:15 STEP 2096 H=2096:00 +CPU= 25.968
  17:34:54 STEP 2097 H=2097:00 +CPU= 38.448
  17:35:19 STEP 2098 H=2098:00 +CPU= 25.284
  17:35:44 STEP 2099 H=2099:00 +CPU= 25.177
  17:36:27 STEP 2100 H=2100:00 +CPU= 42.925
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002100
  17:36:52 STEP 2101 H=2101:00 +CPU= 25.160
  17:37:19 STEP 2102 H=2102:00 +CPU= 26.161
  17:37:58 STEP 2103 H=2103:00 +CPU= 38.955
  17:38:24 STEP 2104 H=2104:00 +CPU= 26.308
  17:38:49 STEP 2105 H=2105:00 +CPU= 25.613
  17:39:30 STEP 2106 H=2106:00 +CPU= 40.880
  17:39:56 STEP 2107 H=2107:00 +CPU= 25.550
  17:40:22 STEP 2108 H=2108:00 +CPU= 26.393
  17:41:02 STEP 2109 H=2109:00 +CPU= 39.703
  17:41:28 STEP 2110 H=2110:00 +CPU= 25.925
  17:41:54 STEP 2111 H=2111:00 +CPU= 25.624
  17:42:40 STEP 2112 H=2112:00 +CPU= 46.754
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_87.zip
Uploading the intermediate file: upload_file_87.zip
  17:43:06 STEP 2113 H=2113:00 +CPU= 25.368
  17:43:32 STEP 2114 H=2114:00 +CPU= 25.698
  17:44:11 STEP 2115 H=2115:00 +CPU= 39.843
  17:44:37 STEP 2116 H=2116:00 +CPU= 25.900
  17:45:02 STEP 2117 H=2117:00 +CPU= 24.831
  17:45:42 STEP 2118 H=2118:00 +CPU= 40.030
  17:46:07 STEP 2119 H=2119:00 +CPU= 24.773
  17:46:34 STEP 2120 H=2120:00 +CPU= 26.689
  17:47:13 STEP 2121 H=2121:00 +CPU= 39.561
  17:47:39 STEP 2122 H=2122:00 +CPU= 25.593
  17:48:04 STEP 2123 H=2123:00 +CPU= 25.636
  17:48:46 STEP 2124 H=2124:00 +CPU= 41.143
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002124
  17:49:09 STEP 2125 H=2125:00 +CPU= 23.247
  17:49:33 STEP 2126 H=2126:00 +CPU= 24.341
  17:50:10 STEP 2127 H=2127:00 +CPU= 36.565
  17:50:36 STEP 2128 H=2128:00 +CPU= 26.281
  17:51:01 STEP 2129 H=2129:00 +CPU= 25.144
  17:51:38 STEP 2130 H=2130:00 +CPU= 37.021
  17:52:03 STEP 2131 H=2131:00 +CPU= 24.682
  17:52:29 STEP 2132 H=2132:00 +CPU= 25.721
  17:53:07 STEP 2133 H=2133:00 +CPU= 38.512
  17:53:32 STEP 2134 H=2134:00 +CPU= 25.193
  17:53:55 STEP 2135 H=2135:00 +CPU= 22.737
  17:54:40 STEP 2136 H=2136:00 +CPU= 44.972
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_88.zip
Uploading the intermediate file: upload_file_88.zip
  17:55:05 STEP 2137 H=2137:00 +CPU= 25.102
  17:55:32 STEP 2138 H=2138:00 +CPU= 26.354
  17:56:11 STEP 2139 H=2139:00 +CPU= 39.134
  17:56:37 STEP 2140 H=2140:00 +CPU= 26.118
  17:57:02 STEP 2141 H=2141:00 +CPU= 24.892
  17:57:43 STEP 2142 H=2142:00 +CPU= 40.698
  17:58:07 STEP 2143 H=2143:00 +CPU= 24.813
  17:58:34 STEP 2144 H=2144:00 +CPU= 26.289
  17:59:13 STEP 2145 H=2145:00 +CPU= 39.542
  17:59:39 STEP 2146 H=2146:00 +CPU= 25.695
  18:00:05 STEP 2147 H=2147:00 +CPU= 25.721
  18:00:48 STEP 2148 H=2148:00 +CPU= 43.618
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002148
  18:01:13 STEP 2149 H=2149:00 +CPU= 24.813
  18:01:39 STEP 2150 H=2150:00 +CPU= 26.405
  18:02:16 STEP 2151 H=2151:00 +CPU= 36.916
  18:02:42 STEP 2152 H=2152:00 +CPU= 25.073
  18:03:07 STEP 2153 H=2153:00 +CPU= 25.169
  18:03:46 STEP 2154 H=2154:00 +CPU= 39.653
  18:04:12 STEP 2155 H=2155:00 +CPU= 25.664
  18:04:38 STEP 2156 H=2156:00 +CPU= 26.164
  18:05:17 STEP 2157 H=2157:00 +CPU= 38.849
  18:05:44 STEP 2158 H=2158:00 +CPU= 26.723
  18:06:10 STEP 2159 H=2159:00 +CPU= 25.967
  18:06:55 STEP 2160 H=2160:00 +CPU= 45.446
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_89.zip
Uploading the intermediate file: upload_file_89.zip
  18:07:19 STEP 2161 H=2161:00 +CPU= 24.188
Uploading trickle at timestep: 7776000
  18:07:44 STEP 2162 H=2162:00 +CPU= 24.818
  18:08:23 STEP 2163 H=2163:00 +CPU= 38.937
  18:08:47 STEP 2164 H=2164:00 +CPU= 23.454
  18:09:11 STEP 2165 H=2165:00 +CPU= 24.468
  18:09:50 STEP 2166 H=2166:00 +CPU= 38.521
  18:10:15 STEP 2167 H=2167:00 +CPU= 25.357
  18:10:40 STEP 2168 H=2168:00 +CPU= 25.261
  18:11:18 STEP 2169 H=2169:00 +CPU= 37.814
  18:11:44 STEP 2170 H=2170:00 +CPU= 26.132
  18:12:10 STEP 2171 H=2171:00 +CPU= 25.531
  18:12:53 STEP 2172 H=2172:00 +CPU= 42.734
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002172
  18:13:18 STEP 2173 H=2173:00 +CPU= 25.799
  18:13:45 STEP 2174 H=2174:00 +CPU= 26.581
  18:14:24 STEP 2175 H=2175:00 +CPU= 39.184
  18:14:50 STEP 2176 H=2176:00 +CPU= 25.616
  18:15:13 STEP 2177 H=2177:00 +CPU= 23.328
  18:15:53 STEP 2178 H=2178:00 +CPU= 39.756
  18:16:19 STEP 2179 H=2179:00 +CPU= 25.620
  18:16:45 STEP 2180 H=2180:00 +CPU= 26.824
  18:17:24 STEP 2181 H=2181:00 +CPU= 38.930
  18:17:52 STEP 2182 H=2182:00 +CPU= 27.348
  18:18:18 STEP 2183 H=2183:00 +CPU= 26.474
  18:19:04 STEP 2184 H=2184:00 +CPU= 45.448
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_90.zip
Uploading the intermediate file: upload_file_90.zip
  18:19:28 STEP 2185 H=2185:00 +CPU= 24.324
  18:19:54 STEP 2186 H=2186:00 +CPU= 26.096
  18:20:31 STEP 2187 H=2187:00 +CPU= 36.195
  18:20:57 STEP 2188 H=2188:00 +CPU= 26.249
  18:21:23 STEP 2189 H=2189:00 +CPU= 25.853
  18:22:04 STEP 2190 H=2190:00 +CPU= 41.446
  18:22:28 STEP 2191 H=2191:00 +CPU= 24.250
  18:22:53 STEP 2192 H=2192:00 +CPU= 24.862
  18:23:32 STEP 2193 H=2193:00 +CPU= 39.229
  18:23:59 STEP 2194 H=2194:00 +CPU= 26.308
  18:24:24 STEP 2195 H=2195:00 +CPU= 25.657
  18:25:07 STEP 2196 H=2196:00 +CPU= 42.969
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002196
  18:25:32 STEP 2197 H=2197:00 +CPU= 25.058
  18:25:59 STEP 2198 H=2198:00 +CPU= 26.787
  18:26:38 STEP 2199 H=2199:00 +CPU= 38.767
  18:27:03 STEP 2200 H=2200:00 +CPU= 24.502
  18:27:28 STEP 2201 H=2201:00 +CPU= 25.737
  18:28:09 STEP 2202 H=2202:00 +CPU= 40.399
  18:28:34 STEP 2203 H=2203:00 +CPU= 24.896
  18:29:00 STEP 2204 H=2204:00 +CPU= 26.112
  18:29:39 STEP 2205 H=2205:00 +CPU= 39.536
  18:30:05 STEP 2206 H=2206:00 +CPU= 26.025
  18:30:32 STEP 2207 H=2207:00 +CPU= 26.502
  18:31:12 STEP 2208 H=2208:00 +CPU= 40.257
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_91.zip
Uploading the intermediate file: upload_file_91.zip
  18:31:37 STEP 2209 H=2209:00 +CPU= 24.703
  18:32:03 STEP 2210 H=2210:00 +CPU= 25.690
  18:32:39 STEP 2211 H=2211:00 +CPU= 36.092
  18:33:03 STEP 2212 H=2212:00 +CPU= 24.033
  18:33:22 STEP 2213 H=2213:00 +CPU= 19.479
  18:33:55 STEP 2214 H=2214:00 +CPU= 32.370
  18:34:16 STEP 2215 H=2215:00 +CPU= 21.293
  18:34:41 STEP 2216 H=2216:00 +CPU= 24.838
  18:35:16 STEP 2217 H=2217:00 +CPU= 34.758
  18:35:40 STEP 2218 H=2218:00 +CPU= 23.977
  18:36:02 STEP 2219 H=2219:00 +CPU= 22.108
  18:36:40 STEP 2220 H=2220:00 +CPU= 38.427
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002220
  18:37:03 STEP 2221 H=2221:00 +CPU= 22.825
  18:37:25 STEP 2222 H=2222:00 +CPU= 22.504
  18:38:03 STEP 2223 H=2223:00 +CPU= 37.507
  18:38:29 STEP 2224 H=2224:00 +CPU= 26.450
  18:38:52 STEP 2225 H=2225:00 +CPU= 22.684
  18:39:31 STEP 2226 H=2226:00 +CPU= 38.625
  18:39:56 STEP 2227 H=2227:00 +CPU= 25.704
  18:40:23 STEP 2228 H=2228:00 +CPU= 26.600
  18:41:02 STEP 2229 H=2229:00 +CPU= 38.746
  18:41:28 STEP 2230 H=2230:00 +CPU= 26.653
  18:41:56 STEP 2231 H=2231:00 +CPU= 27.206
  18:42:41 STEP 2232 H=2232:00 +CPU= 45.012
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_92.zip
Uploading the intermediate file: upload_file_92.zip
  18:43:03 STEP 2233 H=2233:00 +CPU= 22.674
  18:43:29 STEP 2234 H=2234:00 +CPU= 25.935
  18:44:08 STEP 2235 H=2235:00 +CPU= 38.833
  18:44:34 STEP 2236 H=2236:00 +CPU= 26.218
  18:45:00 STEP 2237 H=2237:00 +CPU= 25.754
  18:45:40 STEP 2238 H=2238:00 +CPU= 39.967
  18:46:06 STEP 2239 H=2239:00 +CPU= 25.564
  18:46:32 STEP 2240 H=2240:00 +CPU= 26.637
  18:47:11 STEP 2241 H=2241:00 +CPU= 38.470
  18:47:37 STEP 2242 H=2242:00 +CPU= 26.411
  18:48:03 STEP 2243 H=2243:00 +CPU= 26.139
  18:48:48 STEP 2244 H=2244:00 +CPU= 44.266
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002244
  18:49:14 STEP 2245 H=2245:00 +CPU= 25.945
  18:49:39 STEP 2246 H=2246:00 +CPU= 25.819
  18:50:18 STEP 2247 H=2247:00 +CPU= 38.822
  18:50:43 STEP 2248 H=2248:00 +CPU= 24.701
  18:51:07 STEP 2249 H=2249:00 +CPU= 24.076
  18:51:48 STEP 2250 H=2250:00 +CPU= 40.552
  18:52:13 STEP 2251 H=2251:00 +CPU= 25.416
  18:52:39 STEP 2252 H=2252:00 +CPU= 26.224
  18:53:19 STEP 2253 H=2253:00 +CPU= 39.376
  18:53:45 STEP 2254 H=2254:00 +CPU= 26.314
  18:54:11 STEP 2255 H=2255:00 +CPU= 25.626
  18:54:52 STEP 2256 H=2256:00 +CPU= 41.736
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_93.zip
Uploading the intermediate file: upload_file_93.zip
  18:55:18 STEP 2257 H=2257:00 +CPU= 25.291
  18:55:43 STEP 2258 H=2258:00 +CPU= 25.327
  18:56:20 STEP 2259 H=2259:00 +CPU= 36.358
  18:56:46 STEP 2260 H=2260:00 +CPU= 26.673
  18:57:11 STEP 2261 H=2261:00 +CPU= 24.739
  18:57:50 STEP 2262 H=2262:00 +CPU= 38.583
  18:58:14 STEP 2263 H=2263:00 +CPU= 24.037
  18:58:40 STEP 2264 H=2264:00 +CPU= 25.888
  18:59:18 STEP 2265 H=2265:00 +CPU= 38.469
  18:59:45 STEP 2266 H=2266:00 +CPU= 26.586
  19:00:08 STEP 2267 H=2267:00 +CPU= 23.822
  19:00:49 STEP 2268 H=2268:00 +CPU= 40.555
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002268
  19:01:15 STEP 2269 H=2269:00 +CPU= 25.885
  19:01:42 STEP 2270 H=2270:00 +CPU= 26.691
  19:02:21 STEP 2271 H=2271:00 +CPU= 38.994
  19:02:47 STEP 2272 H=2272:00 +CPU= 26.227
  19:03:13 STEP 2273 H=2273:00 +CPU= 25.945
  19:03:54 STEP 2274 H=2274:00 +CPU= 40.689
  19:04:20 STEP 2275 H=2275:00 +CPU= 26.514
  19:04:46 STEP 2276 H=2276:00 +CPU= 26.352
  19:05:26 STEP 2277 H=2277:00 +CPU= 39.324
  19:05:50 STEP 2278 H=2278:00 +CPU= 24.247
  19:06:15 STEP 2279 H=2279:00 +CPU= 25.206
  19:06:59 STEP 2280 H=2280:00 +CPU= 44.005
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_94.zip
Uploading the intermediate file: upload_file_94.zip
  19:07:25 STEP 2281 H=2281:00 +CPU= 25.630
  19:07:51 STEP 2282 H=2282:00 +CPU= 26.079
  19:08:31 STEP 2283 H=2283:00 +CPU= 39.683
  19:08:57 STEP 2284 H=2284:00 +CPU= 26.056
  19:09:22 STEP 2285 H=2285:00 +CPU= 24.877
  19:10:00 STEP 2286 H=2286:00 +CPU= 38.486
  19:10:26 STEP 2287 H=2287:00 +CPU= 25.548
  19:10:52 STEP 2288 H=2288:00 +CPU= 26.270
  19:11:29 STEP 2289 H=2289:00 +CPU= 36.837
  19:11:53 STEP 2290 H=2290:00 +CPU= 23.503
  19:12:15 STEP 2291 H=2291:00 +CPU= 22.095
  19:12:58 STEP 2292 H=2292:00 +CPU= 43.305
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002292
  19:13:24 STEP 2293 H=2293:00 +CPU= 25.790
  19:13:50 STEP 2294 H=2294:00 +CPU= 26.330
  19:14:29 STEP 2295 H=2295:00 +CPU= 38.679
  19:14:54 STEP 2296 H=2296:00 +CPU= 25.057
  19:15:18 STEP 2297 H=2297:00 +CPU= 24.372
  19:15:57 STEP 2298 H=2298:00 +CPU= 38.761
  19:16:22 STEP 2299 H=2299:00 +CPU= 24.631
  19:16:48 STEP 2300 H=2300:00 +CPU= 25.465
  19:17:24 STEP 2301 H=2301:00 +CPU= 36.926
  19:17:49 STEP 2302 H=2302:00 +CPU= 24.223
  19:18:14 STEP 2303 H=2303:00 +CPU= 25.174
  19:19:00 STEP 2304 H=2304:00 +CPU= 45.763
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833_95.zip
Uploading the intermediate file: upload_file_95.zip
  19:19:25 STEP 2305 H=2305:00 +CPU= 25.401
  19:19:51 STEP 2306 H=2306:00 +CPU= 26.267
  19:20:31 STEP 2307 H=2307:00 +CPU= 39.722
  19:20:57 STEP 2308 H=2308:00 +CPU= 26.297
  19:21:22 STEP 2309 H=2309:00 +CPU= 24.060
  19:22:01 STEP 2310 H=2310:00 +CPU= 39.649
  19:22:28 STEP 2311 H=2311:00 +CPU= 26.599
  19:22:54 STEP 2312 H=2312:00 +CPU= 26.414
  19:23:33 STEP 2313 H=2313:00 +CPU= 39.019
  19:23:59 STEP 2314 H=2314:00 +CPU= 26.090
  19:24:25 STEP 2315 H=2315:00 +CPU= 25.067
  19:25:10 STEP 2316 H=2316:00 +CPU= 45.582
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002316
  19:25:37 STEP 2317 H=2317:00 +CPU= 26.645
  19:26:03 STEP 2318 H=2318:00 +CPU= 25.903
  19:26:42 STEP 2319 H=2319:00 +CPU= 39.518
  19:27:08 STEP 2320 H=2320:00 +CPU= 26.062
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4745MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4745 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192716:1671910036:69302.020] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192717:1671910037:69303.020] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192717:1671910037:69303.021] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192717:1671910037:69303.021] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1796] Returned from previous signal handler (at 0x1ce8bb0, signal#15, time taken = 0s), nsigs = 1
[EC_DRHOOK:c1:1:1:5433:5433] [20221224:192717:1671910037:69303.021] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1841] Error _exit(143) upon receipt of signal#15, nsigs = 1
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0561
(argv4) batchid: 979
(argv5) wuid: 12196205
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/12
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12196205
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_83848ea987a78dddb944dee8bc77ae0d to: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0561_2010050100_123_979_12196205.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/12/oifs_43r3_ps_0561_2010050100_123_979_12196205.zip
ic_ancil_file: ic_ancil_12196205
ifsdata_file: ifsdata_12196205
climate_data_file: clim_data_12196205
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1ec658b420363a26f75fb37b9b0fde35 to: /var/lib/boinc-client/slots/12/ic_ancil_12196205.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/12/ic_ancil_12196205.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8f2dd3c019eb1c754353b7f1ca3d32cc to: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12196205.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/12/ifsdata/ifsdata_12196205.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/12/159l_2/clim_data_12196205.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/12/159l_2/clim_data_12196205.zip
Checking for progress XML file: /var/lib/boinc-client/slots/12/progress_file_12196205.xml
Opened progress file ok : /var/lib/boinc-client/slots/12/progress_file_12196205.xml
-- Model is restarting --
Adjusting last_iter, 2321, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/12/progress_file_12196205.xml
last_cpu_time: 69255
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0561_2010050100_123_979_12196205_0_r171063833
The child process has been launched with process id: 1151
Executing the command: /var/lib/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:185249:1673463169:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:54:42 STEP 2304 H=2304:00 +CPU= 48.651
  18:55:10 STEP 2305 H=2305:00 +CPU= 27.668
  18:55:35 STEP 2306 H=2306:00 +CPU= 24.822
  18:56:16 STEP 2307 H=2307:00 +CPU= 40.949
  18:56:42 STEP 2308 H=2308:00 +CPU= 26.347
  18:57:07 STEP 2309 H=2309:00 +CPU= 24.885
  18:57:51 STEP 2310 H=2310:00 +CPU= 43.794
  18:58:19 STEP 2311 H=2311:00 +CPU= 28.094
  18:58:46 STEP 2312 H=2312:00 +CPU= 26.512
  18:59:28 STEP 2313 H=2313:00 +CPU= 42.006
  18:59:54 STEP 2314 H=2314:00 +CPU= 26.143
  19:00:21 STEP 2315 H=2315:00 +CPU= 27.200
  19:01:09 STEP 2316 H=2316:00 +CPU= 47.931
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/12/ICMUAhq0f+002316
  19:01:37 STEP 2317 H=2317:00 +CPU= 27.347
  19:02:03 STEP 2318 H=2318:00 +CPU= 25.894
  19:02:46 STEP 2319 H=2319:00 +CPU= 43.018
  19:03:14 STEP 2320 H=2320:00 +CPU= 28.788
  19:03:42 STEP 2321 H=2321:00 +CPU= 27.419
  19:04:24 STEP 2322 H=2322:00 +CPU= 42.495
  19:04:52 STEP 2323 H=2323:00 +CPU= 27.557
  19:05:20 STEP 2324 H=2324:00 +CPU= 27.729
  19:06:02 STEP 2325 H=2325:00 +CPU= 42.500
  19:06:30 STEP 2326 H=2326:00 +CPU= 27.573
  19:07:00 STEP 2327 H=2327:00 +CPU= 29.854
ECCODES ERROR   :  IO ERROR: No space left on device (No space left on device)
 GRIB_WRITE_BYTES_INT        50001  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_WRITE_BYTES_REAL8 FAILED
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2784 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOPACK 
[EC_DRHOOK:c1:1:1:1151:1151] [20230111:190702:1673464022:853.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          WRRESF 
[EC_DRHOOK:c1:1:1:1151:1151] [..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:52:56 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 7,776,000 64,444 0.0083
09 May 2023 15:52:43 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 6,912,000 57,216 0.0083
09 May 2023 15:52:31 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 6,048,000 49,998 0.0083
09 May 2023 15:52:19 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 5,184,000 42,781 0.0083
09 May 2023 15:52:08 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 4,320,000 35,593 0.0082
09 May 2023 15:51:56 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 3,456,000 28,368 0.0082
09 May 2023 15:51:46 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 2,592,000 21,183 0.0082
09 May 2023 15:51:35 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 1,728,000 13,976 0.0081
09 May 2023 15:51:24 1537350 22282166 oifs_43r3_ps_0561_2010050100_123_979_12196205_0 864,000 6,769 0.0078


©2024 climateprediction.net