climateprediction.net home page
Task 22293843

Task 22293843

Name oifs_43r3_ps_0252_2000050100_123_969_12185896_1
Workunit 12185896
Created 27 Dec 2022, 7:34:45 UTC
Sent 18 Jan 2023, 19:37:07 UTC
Report deadline 19 Mar 2023, 19:37:07 UTC
Received 19 Jan 2023, 21:47:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538236
Run time 16 hours 55 min 56 sec
CPU time 16 hours 45 min 59 sec
Validate state Invalid
Credit 11,007.36
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.88 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,655.09 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
P 1937 H=1937:00 +CPU= 21.181
  18:11:34 STEP 1938 H=1938:00 +CPU= 30.648
  18:11:57 STEP 1939 H=1939:00 +CPU= 22.399
  18:12:21 STEP 1940 H=1940:00 +CPU= 24.499
  18:12:56 STEP 1941 H=1941:00 +CPU= 35.191
  18:13:18 STEP 1942 H=1942:00 +CPU= 21.634
  18:13:39 STEP 1943 H=1943:00 +CPU= 21.322
  18:14:20 STEP 1944 H=1944:00 +CPU= 41.002
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_80.zip
Uploading the intermediate file: upload_file_80.zip
  18:14:41 STEP 1945 H=1945:00 +CPU= 21.015
Uploading trickle at timestep: 6998400
  18:15:04 STEP 1946 H=1946:00 +CPU= 22.866
  18:15:37 STEP 1947 H=1947:00 +CPU= 32.500
  18:15:57 STEP 1948 H=1948:00 +CPU= 20.773
  18:16:19 STEP 1949 H=1949:00 +CPU= 21.408
  18:16:56 STEP 1950 H=1950:00 +CPU= 37.004
  18:17:20 STEP 1951 H=1951:00 +CPU= 24.210
  18:17:43 STEP 1952 H=1952:00 +CPU= 22.725
  18:18:17 STEP 1953 H=1953:00 +CPU= 34.408
  18:18:37 STEP 1954 H=1954:00 +CPU= 19.903
  18:19:00 STEP 1955 H=1955:00 +CPU= 22.697
  18:19:36 STEP 1956 H=1956:00 +CPU= 35.721
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+001956
  18:19:58 STEP 1957 H=1957:00 +CPU= 21.963
  18:20:20 STEP 1958 H=1958:00 +CPU= 22.206
  18:20:51 STEP 1959 H=1959:00 +CPU= 31.640
  18:21:14 STEP 1960 H=1960:00 +CPU= 22.174
  18:21:35 STEP 1961 H=1961:00 +CPU= 21.544
  18:22:08 STEP 1962 H=1962:00 +CPU= 33.145
  18:22:29 STEP 1963 H=1963:00 +CPU= 20.459
  18:22:52 STEP 1964 H=1964:00 +CPU= 23.397
  18:23:28 STEP 1965 H=1965:00 +CPU= 35.891
  18:23:49 STEP 1966 H=1966:00 +CPU= 21.390
  18:24:09 STEP 1967 H=1967:00 +CPU= 20.046
  18:24:50 STEP 1968 H=1968:00 +CPU= 40.485
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_81.zip
Uploading the intermediate file: upload_file_81.zip
  18:25:14 STEP 1969 H=1969:00 +CPU= 23.985
  18:25:37 STEP 1970 H=1970:00 +CPU= 22.775
  18:26:09 STEP 1971 H=1971:00 +CPU= 32.613
  18:26:30 STEP 1972 H=1972:00 +CPU= 20.858
  18:26:50 STEP 1973 H=1973:00 +CPU= 19.776
  18:27:26 STEP 1974 H=1974:00 +CPU= 36.072
  18:27:45 STEP 1975 H=1975:00 +CPU= 19.287
  18:28:06 STEP 1976 H=1976:00 +CPU= 20.663
  18:28:41 STEP 1977 H=1977:00 +CPU= 34.544
  18:29:05 STEP 1978 H=1978:00 +CPU= 24.532
  18:29:26 STEP 1979 H=1979:00 +CPU= 20.420
  18:30:04 STEP 1980 H=1980:00 +CPU= 38.213
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+001980
  18:30:24 STEP 1981 H=1981:00 +CPU= 20.675
  18:30:49 STEP 1982 H=1982:00 +CPU= 24.724
  18:31:24 STEP 1983 H=1983:00 +CPU= 34.743
  18:31:46 STEP 1984 H=1984:00 +CPU= 21.928
  18:32:06 STEP 1985 H=1985:00 +CPU= 19.929
  18:32:36 STEP 1986 H=1986:00 +CPU= 30.520
  18:32:57 STEP 1987 H=1987:00 +CPU= 20.523
  18:33:22 STEP 1988 H=1988:00 +CPU= 24.997
  18:33:56 STEP 1989 H=1989:00 +CPU= 33.685
  18:34:21 STEP 1990 H=1990:00 +CPU= 25.208
  18:34:46 STEP 1991 H=1991:00 +CPU= 25.211
  18:35:31 STEP 1992 H=1992:00 +CPU= 44.798
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_82.zip
Uploading the intermediate file: upload_file_82.zip
  18:35:55 STEP 1993 H=1993:00 +CPU= 24.428
  18:36:18 STEP 1994 H=1994:00 +CPU= 22.718
  18:36:51 STEP 1995 H=1995:00 +CPU= 33.011
  18:37:14 STEP 1996 H=1996:00 +CPU= 22.999
  18:37:36 STEP 1997 H=1997:00 +CPU= 21.798
  18:38:11 STEP 1998 H=1998:00 +CPU= 35.672
  18:38:34 STEP 1999 H=1999:00 +CPU= 22.832
  18:38:57 STEP 2000 H=2000:00 +CPU= 22.421
  18:39:30 STEP 2001 H=2001:00 +CPU= 33.647
  18:39:51 STEP 2002 H=2002:00 +CPU= 20.424
  18:40:13 STEP 2003 H=2003:00 +CPU= 22.032
  18:40:47 STEP 2004 H=2004:00 +CPU= 33.640
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002004
  18:41:08 STEP 2005 H=2005:00 +CPU= 21.678
  18:41:31 STEP 2006 H=2006:00 +CPU= 22.995
  18:42:08 STEP 2007 H=2007:00 +CPU= 37.124
  18:42:31 STEP 2008 H=2008:00 +CPU= 22.475
  18:42:53 STEP 2009 H=2009:00 +CPU= 21.771
  18:43:23 STEP 2010 H=2010:00 +CPU= 30.050
  18:43:42 STEP 2011 H=2011:00 +CPU= 19.439
  18:44:07 STEP 2012 H=2012:00 +CPU= 24.632
  18:44:41 STEP 2013 H=2013:00 +CPU= 34.402
  18:45:02 STEP 2014 H=2014:00 +CPU= 20.638
  18:45:25 STEP 2015 H=2015:00 +CPU= 23.562
  18:46:05 STEP 2016 H=2016:00 +CPU= 39.970
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_83.zip
Uploading the intermediate file: upload_file_83.zip
  18:46:30 STEP 2017 H=2017:00 +CPU= 24.536
  18:46:54 STEP 2018 H=2018:00 +CPU= 24.157
  18:47:29 STEP 2019 H=2019:00 +CPU= 35.036
  18:47:52 STEP 2020 H=2020:00 +CPU= 22.776
  18:48:14 STEP 2021 H=2021:00 +CPU= 21.920
  18:48:49 STEP 2022 H=2022:00 +CPU= 35.322
  18:49:09 STEP 2023 H=2023:00 +CPU= 19.731
  18:49:32 STEP 2024 H=2024:00 +CPU= 22.929
  18:50:06 STEP 2025 H=2025:00 +CPU= 34.202
  18:50:30 STEP 2026 H=2026:00 +CPU= 24.069
  18:50:51 STEP 2027 H=2027:00 +CPU= 20.771
  18:51:30 STEP 2028 H=2028:00 +CPU= 39.344
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002028
  18:51:53 STEP 2029 H=2029:00 +CPU= 22.584
  18:52:17 STEP 2030 H=2030:00 +CPU= 24.597
  18:52:51 STEP 2031 H=2031:00 +CPU= 33.581
  18:53:11 STEP 2032 H=2032:00 +CPU= 19.700
  18:53:33 STEP 2033 H=2033:00 +CPU= 22.783
  18:54:09 STEP 2034 H=2034:00 +CPU= 35.174
  18:54:28 STEP 2035 H=2035:00 +CPU= 19.554
  18:54:50 STEP 2036 H=2036:00 +CPU= 21.722
  18:55:24 STEP 2037 H=2037:00 +CPU= 34.279
  18:55:42 STEP 2038 H=2038:00 +CPU= 17.717
  18:56:04 STEP 2039 H=2039:00 +CPU= 22.138
  18:56:46 STEP 2040 H=2040:00 +CPU= 41.727
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_84.zip
Uploading the intermediate file: upload_file_84.zip
  18:57:08 STEP 2041 H=2041:00 +CPU= 22.145
  18:57:30 STEP 2042 H=2042:00 +CPU= 21.550
  18:58:04 STEP 2043 H=2043:00 +CPU= 34.455
  18:58:22 STEP 2044 H=2044:00 +CPU= 18.302
  18:58:42 STEP 2045 H=2045:00 +CPU= 19.849
  18:59:18 STEP 2046 H=2046:00 +CPU= 35.421
  18:59:41 STEP 2047 H=2047:00 +CPU= 23.125
  19:00:05 STEP 2048 H=2048:00 +CPU= 24.134
  19:00:37 STEP 2049 H=2049:00 +CPU= 31.787
  19:01:02 STEP 2050 H=2050:00 +CPU= 25.220
  19:01:26 STEP 2051 H=2051:00 +CPU= 24.468
  19:02:05 STEP 2052 H=2052:00 +CPU= 38.727
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002052
  19:02:27 STEP 2053 H=2053:00 +CPU= 22.063
  19:02:51 STEP 2054 H=2054:00 +CPU= 23.447
  19:03:29 STEP 2055 H=2055:00 +CPU= 37.780
  19:03:51 STEP 2056 H=2056:00 +CPU= 21.998
  19:04:13 STEP 2057 H=2057:00 +CPU= 22.473
  19:04:52 STEP 2058 H=2058:00 +CPU= 39.464
  19:05:17 STEP 2059 H=2059:00 +CPU= 24.462
  19:05:42 STEP 2060 H=2060:00 +CPU= 25.386
  19:06:18 STEP 2061 H=2061:00 +CPU= 35.300
  19:06:42 STEP 2062 H=2062:00 +CPU= 24.364
  19:07:05 STEP 2063 H=2063:00 +CPU= 22.773
  19:07:44 STEP 2064 H=2064:00 +CPU= 38.796
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_85.zip
Uploading the intermediate file: upload_file_85.zip
  19:08:05 STEP 2065 H=2065:00 +CPU= 21.834
  19:08:27 STEP 2066 H=2066:00 +CPU= 21.610
  19:08:58 STEP 2067 H=2067:00 +CPU= 30.665
  19:09:19 STEP 2068 H=2068:00 +CPU= 21.376
  19:09:35 STEP 2069 H=2069:00 +CPU= 15.974
  19:10:08 STEP 2070 H=2070:00 +CPU= 32.569
  19:10:32 STEP 2071 H=2071:00 +CPU= 23.822
  19:10:57 STEP 2072 H=2072:00 +CPU= 24.961
  19:11:32 STEP 2073 H=2073:00 +CPU= 35.028
  19:11:55 STEP 2074 H=2074:00 +CPU= 23.299
  19:12:18 STEP 2075 H=2075:00 +CPU= 22.977
  19:12:55 STEP 2076 H=2076:00 +CPU= 37.503
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002076
  19:13:19 STEP 2077 H=2077:00 +CPU= 23.353
  19:13:39 STEP 2078 H=2078:00 +CPU= 20.132
  19:14:17 STEP 2079 H=2079:00 +CPU= 37.757
  19:14:44 STEP 2080 H=2080:00 +CPU= 27.310
  19:15:08 STEP 2081 H=2081:00 +CPU= 23.712
  19:15:45 STEP 2082 H=2082:00 +CPU= 37.421
  19:16:07 STEP 2083 H=2083:00 +CPU= 22.254
  19:16:31 STEP 2084 H=2084:00 +CPU= 23.722
  19:17:05 STEP 2085 H=2085:00 +CPU= 33.483
  19:17:29 STEP 2086 H=2086:00 +CPU= 24.605
  19:17:51 STEP 2087 H=2087:00 +CPU= 21.427
  19:18:30 STEP 2088 H=2088:00 +CPU= 39.651
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_86.zip
Uploading the intermediate file: upload_file_86.zip
  19:18:50 STEP 2089 H=2089:00 +CPU= 19.873
  19:19:12 STEP 2090 H=2090:00 +CPU= 21.542
  19:19:47 STEP 2091 H=2091:00 +CPU= 35.546
  19:20:09 STEP 2092 H=2092:00 +CPU= 21.520
  19:20:32 STEP 2093 H=2093:00 +CPU= 23.056
  19:21:04 STEP 2094 H=2094:00 +CPU= 32.349
  19:21:27 STEP 2095 H=2095:00 +CPU= 23.157
  19:21:49 STEP 2096 H=2096:00 +CPU= 21.749
  19:22:24 STEP 2097 H=2097:00 +CPU= 34.764
  19:22:44 STEP 2098 H=2098:00 +CPU= 20.080
  19:23:08 STEP 2099 H=2099:00 +CPU= 23.694
  19:23:43 STEP 2100 H=2100:00 +CPU= 35.271
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002100
  19:24:05 STEP 2101 H=2101:00 +CPU= 21.736
  19:24:27 STEP 2102 H=2102:00 +CPU= 22.522
  19:25:00 STEP 2103 H=2103:00 +CPU= 32.598
  19:25:26 STEP 2104 H=2104:00 +CPU= 26.096
  19:25:51 STEP 2105 H=2105:00 +CPU= 24.678
  19:26:29 STEP 2106 H=2106:00 +CPU= 38.616
  19:26:53 STEP 2107 H=2107:00 +CPU= 24.214
  19:27:18 STEP 2108 H=2108:00 +CPU= 24.438
  19:27:54 STEP 2109 H=2109:00 +CPU= 35.741
  19:28:18 STEP 2110 H=2110:00 +CPU= 24.787
  19:28:43 STEP 2111 H=2111:00 +CPU= 24.459
  19:29:25 STEP 2112 H=2112:00 +CPU= 42.228
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_87.zip
Uploading the intermediate file: upload_file_87.zip
  19:29:45 STEP 2113 H=2113:00 +CPU= 20.185
  19:30:07 STEP 2114 H=2114:00 +CPU= 21.780
  19:30:42 STEP 2115 H=2115:00 +CPU= 35.336
  19:31:07 STEP 2116 H=2116:00 +CPU= 24.769
  19:31:26 STEP 2117 H=2117:00 +CPU= 18.403
  19:32:05 STEP 2118 H=2118:00 +CPU= 39.319
  19:32:29 STEP 2119 H=2119:00 +CPU= 23.817
  19:32:53 STEP 2120 H=2120:00 +CPU= 24.636
  19:33:30 STEP 2121 H=2121:00 +CPU= 37.051
  19:33:51 STEP 2122 H=2122:00 +CPU= 20.673
  19:34:13 STEP 2123 H=2123:00 +CPU= 22.231
  19:34:48 STEP 2124 H=2124:00 +CPU= 34.706
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002124
  19:35:10 STEP 2125 H=2125:00 +CPU= 22.020
  19:35:29 STEP 2126 H=2126:00 +CPU= 19.153
  19:36:02 STEP 2127 H=2127:00 +CPU= 32.848
  19:36:23 STEP 2128 H=2128:00 +CPU= 20.649
  19:36:44 STEP 2129 H=2129:00 +CPU= 21.020
  19:37:18 STEP 2130 H=2130:00 +CPU= 34.074
  19:37:42 STEP 2131 H=2131:00 +CPU= 24.173
  19:38:05 STEP 2132 H=2132:00 +CPU= 23.220
  19:38:40 STEP 2133 H=2133:00 +CPU= 34.281
  19:39:00 STEP 2134 H=2134:00 +CPU= 20.048
  19:39:22 STEP 2135 H=2135:00 +CPU= 22.774
  19:40:03 STEP 2136 H=2136:00 +CPU= 40.203
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_88.zip
Uploading the intermediate file: upload_file_88.zip
  19:40:26 STEP 2137 H=2137:00 +CPU= 23.347
  19:40:49 STEP 2138 H=2138:00 +CPU= 22.519
  19:41:23 STEP 2139 H=2139:00 +CPU= 34.459
  19:41:43 STEP 2140 H=2140:00 +CPU= 20.239
  19:42:03 STEP 2141 H=2141:00 +CPU= 20.230
  19:42:34 STEP 2142 H=2142:00 +CPU= 30.850
  19:42:56 STEP 2143 H=2143:00 +CPU= 21.719
  19:43:20 STEP 2144 H=2144:00 +CPU= 23.690
  19:43:55 STEP 2145 H=2145:00 +CPU= 34.842
  19:44:16 STEP 2146 H=2146:00 +CPU= 21.195
  19:44:38 STEP 2147 H=2147:00 +CPU= 21.854
  19:45:16 STEP 2148 H=2148:00 +CPU= 38.474
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002148
  19:45:38 STEP 2149 H=2149:00 +CPU= 22.242
  19:46:01 STEP 2150 H=2150:00 +CPU= 23.004
  19:46:35 STEP 2151 H=2151:00 +CPU= 33.315
  19:46:55 STEP 2152 H=2152:00 +CPU= 20.011
  19:47:14 STEP 2153 H=2153:00 +CPU= 19.596
  19:47:47 STEP 2154 H=2154:00 +CPU= 32.367
  19:48:09 STEP 2155 H=2155:00 +CPU= 21.745
  19:48:30 STEP 2156 H=2156:00 +CPU= 21.033
  19:49:05 STEP 2157 H=2157:00 +CPU= 35.471
  19:49:26 STEP 2158 H=2158:00 +CPU= 20.642
  19:49:48 STEP 2159 H=2159:00 +CPU= 22.361
  19:50:29 STEP 2160 H=2160:00 +CPU= 40.880
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_89.zip
Uploading the intermediate file: upload_file_89.zip
  19:50:52 STEP 2161 H=2161:00 +CPU= 22.624
  19:51:12 STEP 2162 H=2162:00 +CPU= 20.417
  19:51:45 STEP 2163 H=2163:00 +CPU= 32.621
  19:52:05 STEP 2164 H=2164:00 +CPU= 20.478
  19:52:28 STEP 2165 H=2165:00 +CPU= 22.475
  19:53:03 STEP 2166 H=2166:00 +CPU= 35.331
  19:53:24 STEP 2167 H=2167:00 +CPU= 21.486
  19:53:43 STEP 2168 H=2168:00 +CPU= 18.058
  19:54:17 STEP 2169 H=2169:00 +CPU= 34.033
  19:54:37 STEP 2170 H=2170:00 +CPU= 20.055
  19:54:59 STEP 2171 H=2171:00 +CPU= 22.040
  19:55:37 STEP 2172 H=2172:00 +CPU= 38.649
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002172
  19:55:59 STEP 2173 H=2173:00 +CPU= 21.652
  19:56:21 STEP 2174 H=2174:00 +CPU= 22.151
  19:56:56 STEP 2175 H=2175:00 +CPU= 34.875
  19:57:20 STEP 2176 H=2176:00 +CPU= 24.366
  19:57:42 STEP 2177 H=2177:00 +CPU= 21.967
  19:58:18 STEP 2178 H=2178:00 +CPU= 35.610
  19:58:38 STEP 2179 H=2179:00 +CPU= 20.007
  19:59:00 STEP 2180 H=2180:00 +CPU= 22.022
  19:59:31 STEP 2181 H=2181:00 +CPU= 31.245
  19:59:53 STEP 2182 H=2182:00 +CPU= 21.301
  20:00:12 STEP 2183 H=2183:00 +CPU= 19.349
  20:00:53 STEP 2184 H=2184:00 +CPU= 40.968
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_90.zip
Uploading the intermediate file: upload_file_90.zip
  20:01:15 STEP 2185 H=2185:00 +CPU= 22.296
Uploading trickle at timestep: 7862400
  20:01:38 STEP 2186 H=2186:00 +CPU= 22.585
  20:02:10 STEP 2187 H=2187:00 +CPU= 31.793
  20:02:32 STEP 2188 H=2188:00 +CPU= 22.501
  20:02:53 STEP 2189 H=2189:00 +CPU= 21.011
  20:03:25 STEP 2190 H=2190:00 +CPU= 31.874
  20:03:45 STEP 2191 H=2191:00 +CPU= 19.470
  20:04:05 STEP 2192 H=2192:00 +CPU= 20.889
  20:04:37 STEP 2193 H=2193:00 +CPU= 31.351
  20:04:58 STEP 2194 H=2194:00 +CPU= 21.078
  20:05:19 STEP 2195 H=2195:00 +CPU= 21.132
  20:05:57 STEP 2196 H=2196:00 +CPU= 37.449
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002196
  20:06:18 STEP 2197 H=2197:00 +CPU= 20.991
  20:06:39 STEP 2198 H=2198:00 +CPU= 21.490
  20:07:12 STEP 2199 H=2199:00 +CPU= 32.719
  20:07:32 STEP 2200 H=2200:00 +CPU= 20.208
  20:07:54 STEP 2201 H=2201:00 +CPU= 21.783
  20:08:30 STEP 2202 H=2202:00 +CPU= 36.072
  20:08:52 STEP 2203 H=2203:00 +CPU= 21.819
  20:09:13 STEP 2204 H=2204:00 +CPU= 21.461
  20:09:45 STEP 2205 H=2205:00 +CPU= 31.793
  20:10:07 STEP 2206 H=2206:00 +CPU= 21.868
  20:10:27 STEP 2207 H=2207:00 +CPU= 20.527
  20:11:06 STEP 2208 H=2208:00 +CPU= 39.182
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_91.zip
Uploading the intermediate file: upload_file_91.zip
  20:11:29 STEP 2209 H=2209:00 +CPU= 22.247
  20:11:51 STEP 2210 H=2210:00 +CPU= 22.104
  20:12:27 STEP 2211 H=2211:00 +CPU= 36.096
  20:12:49 STEP 2212 H=2212:00 +CPU= 22.371
  20:13:07 STEP 2213 H=2213:00 +CPU= 17.376
  20:13:40 STEP 2214 H=2214:00 +CPU= 33.153
  20:14:00 STEP 2215 H=2215:00 +CPU= 19.843
  20:14:23 STEP 2216 H=2216:00 +CPU= 23.094
  20:14:57 STEP 2217 H=2217:00 +CPU= 34.243
  20:15:20 STEP 2218 H=2218:00 +CPU= 22.634
  20:15:40 STEP 2219 H=2219:00 +CPU= 20.734
  20:16:17 STEP 2220 H=2220:00 +CPU= 36.306
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002220
  20:16:37 STEP 2221 H=2221:00 +CPU= 20.044
  20:16:59 STEP 2222 H=2222:00 +CPU= 21.898
  20:17:30 STEP 2223 H=2223:00 +CPU= 31.304
  20:17:52 STEP 2224 H=2224:00 +CPU= 21.899
  20:18:14 STEP 2225 H=2225:00 +CPU= 21.985
  20:18:46 STEP 2226 H=2226:00 +CPU= 31.707
  20:19:06 STEP 2227 H=2227:00 +CPU= 20.021
  20:19:23 STEP 2228 H=2228:00 +CPU= 17.318
  20:19:54 STEP 2229 H=2229:00 +CPU= 31.356
  20:20:17 STEP 2230 H=2230:00 +CPU= 22.444
  20:20:40 STEP 2231 H=2231:00 +CPU= 23.116
  20:21:19 STEP 2232 H=2232:00 +CPU= 39.382
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_92.zip
Uploading the intermediate file: upload_file_92.zip
  20:21:41 STEP 2233 H=2233:00 +CPU= 21.208
  20:22:02 STEP 2234 H=2234:00 +CPU= 21.935
  20:22:38 STEP 2235 H=2235:00 +CPU= 35.448
  20:23:03 STEP 2236 H=2236:00 +CPU= 24.964
  20:23:24 STEP 2237 H=2237:00 +CPU= 21.428
  20:23:59 STEP 2238 H=2238:00 +CPU= 34.517
  20:24:22 STEP 2239 H=2239:00 +CPU= 23.477
  20:24:43 STEP 2240 H=2240:00 +CPU= 20.905
  20:25:19 STEP 2241 H=2241:00 +CPU= 35.731
  20:25:40 STEP 2242 H=2242:00 +CPU= 21.041
  20:25:59 STEP 2243 H=2243:00 +CPU= 19.144
  20:26:33 STEP 2244 H=2244:00 +CPU= 34.187
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002244
  20:26:54 STEP 2245 H=2245:00 +CPU= 20.422
  20:27:14 STEP 2246 H=2246:00 +CPU= 20.471
  20:27:46 STEP 2247 H=2247:00 +CPU= 31.349
  20:28:08 STEP 2248 H=2248:00 +CPU= 22.296
  20:28:31 STEP 2249 H=2249:00 +CPU= 22.697
  20:29:06 STEP 2250 H=2250:00 +CPU= 35.727
  20:29:25 STEP 2251 H=2251:00 +CPU= 18.839
  20:29:47 STEP 2252 H=2252:00 +CPU= 22.187
  20:30:21 STEP 2253 H=2253:00 +CPU= 34.085
  20:30:43 STEP 2254 H=2254:00 +CPU= 21.393
  20:31:07 STEP 2255 H=2255:00 +CPU= 23.978
  20:31:48 STEP 2256 H=2256:00 +CPU= 40.696
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158_93.zip
Uploading the intermediate file: upload_file_93.zip
  20:32:10 STEP 2257 H=2257:00 +CPU= 22.441
  20:32:30 STEP 2258 H=2258:00 +CPU= 19.591
  20:33:05 STEP 2259 H=2259:00 +CPU= 35.378
  20:33:26 STEP 2260 H=2260:00 +CPU= 20.978
  20:33:49 STEP 2261 H=2261:00 +CPU= 23.263
  20:34:27 STEP 2262 H=2262:00 +CPU= 37.578
  20:34:49 STEP 2263 H=2263:00 +CPU= 22.421
  20:35:12 STEP 2264 H=2264:00 +CPU= 22.927
  20:35:48 STEP 2265 H=2265:00 +CPU= 36.072
  20:36:13 STEP 2266 H=2266:00 +CPU= 24.240
  20:36:34 STEP 2267 H=2267:00 +CPU= 21.969
  20:37:13 STEP 2268 H=2268:00 +CPU= 38.732
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client_40000/slots/4/ICMUAhq0f+002268
  20:37:35 STEP 2269 H=2269:00 +CPU= 21.389
  20:37:55 STEP 2270 H=2270:00 +CPU= 20.503
  20:38:27 STEP 2271 H=2271:00 +CPU= 31.687
  20:38:50 STEP 2272 H=2272:00 +CPU= 22.748
  20:39:13 STEP 2273 H=2273:00 +CPU= 22.930
  20:39:45 STEP 2274 H=2274:00 +CPU= 32.348
  20:40:06 STEP 2275 H=2275:00 +CPU= 20.701
  20:40:25 STEP 2276 H=2276:00 +CPU= 19.499
  20:40:58 STEP 2277 H=2277:00 +CPU= 32.761
  20:41:21 STEP 2278 H=2278:00 +CPU= 23.558
  20:41:39 STEP 2279 H=2279:00 +CPU= 17.541
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0252
(argv4) batchid: 969
(argv5) wuid: 12185896
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/4
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0e105172674e761199479aa1637b172b to: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_0252_2000050100_123_969_12185896.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_0252_2000050100_123_969_12185896.zip
ic_ancil_file: ic_ancil_12185896
ifsdata_file: ifsdata_12185896
climate_data_file: clim_data_12185896
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_594c7c10dd6b24c78196befd4a7629c1 to: /var/lib/boinc-client_40000/slots/4/ic_ancil_12185896.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/4/ic_ancil_12185896.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3b8cb718c3a1a81dd1b011f2885a00ec to: /var/lib/boinc-client_40000/slots/4/ifsdata/ifsdata_12185896.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/4/ifsdata/ifsdata_12185896.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/4/159l_2/clim_data_12185896.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/4/159l_2/clim_data_12185896.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
-- Model is restarting --
Adjusting last_iter, 2280, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
last_cpu_time: 60359
upload_file_number: 94
last_iter: 2279
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158
The child process has been launched with process id: 1425
Executing the command: /var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205645:1674161805:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ECCODES ERROR   :  IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device)
 GRIB_READ_BYTES_REAL8        50000  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_READ_BYTES_REAL8 FAILED
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1176 MB (maxheap), 426 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:e1:1:1:1425:1425] [20230119:205652:1674161812:7.038] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7ffdf569b700
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 16) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x8c3229 0xa048b9 0x9e3c9a 0x9dbaf0 0x54b853 0x546a18 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x8c3229] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xa048b9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x9e3c9a] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x9dbaf0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x54b853] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x546a18] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
abort: 
The child process terminated with status: 0
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2000050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0252
(argv4) batchid: 969
(argv5) wuid: 12185896
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/4
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12185896
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0e105172674e761199479aa1637b172b to: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_0252_2000050100_123_969_12185896.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/4/oifs_43r3_ps_0252_2000050100_123_969_12185896.zip
ic_ancil_file: ic_ancil_12185896
ifsdata_file: ifsdata_12185896
climate_data_file: clim_data_12185896
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_594c7c10dd6b24c78196befd4a7629c1 to: /var/lib/boinc-client_40000/slots/4/ic_ancil_12185896.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/4/ic_ancil_12185896.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3b8cb718c3a1a81dd1b011f2885a00ec to: /var/lib/boinc-client_40000/slots/4/ifsdata/ifsdata_12185896.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/4/ifsdata/ifsdata_12185896.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/4/159l_2/clim_data_12185896.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/4/159l_2/clim_data_12185896.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/4/progress_file_12185896.xml
last_cpu_time: 60359
upload_file_number: 94
last_iter: -985
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0252_2000050100_123_969_12185896_1_r476155158
The child process has been launched with process id: 1437
Executing the command: /var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213111:1674163871:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ECCODES ERROR   :  IO ERROR: Inappropriate ioctl for device (Inappropriate ioctl for device)
 GRIB_READ_BYTES_REAL8        50000  FAILED         -11
 GRIB_API ERROR MSG: Input output problem
MPL_ABORT: CALLED FROM PROCESSOR      1 THRD     1
 MPL_ABORT: THRD           1   GRIB_READ_BYTES_REAL8 FAILED
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1176 MB (maxheap), 426 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       RERESF 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        IOSTREAM_MIX:IO_GET 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         IOSTREAM_MIX:READ_RECORD 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          >MPL-IOSTREAM READ_RECORD ( 650) 
[EC_DRHOOK:e1:1:1:1437:1437] [20230119:213118:1674163878:6.922] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GRIB_API:IGRIB_READ_BYTES_REAL8 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7fff39881080
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 16) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client_40000/slots/4/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x8c3229 0xa048b9 0x9e3c9a 0x9dbaf0 0x54b853 0x546a18 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x8c3229] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0xa048b9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x9e3c9a] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x9dbaf0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x54b853] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x546a18] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
abort: 
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:29:19 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 7,862,400 57,909 0.0074
09 May 2023 16:29:06 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 6,998,400 51,515 0.0074
09 May 2023 16:28:55 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 6,134,400 45,273 0.0074
09 May 2023 16:28:49 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 5,270,400 39,081 0.0074
09 May 2023 16:28:38 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 4,406,400 32,828 0.0075
09 May 2023 16:28:27 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 3,542,400 26,566 0.0075
09 May 2023 16:28:04 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 1,814,400 13,882 0.0077
09 May 2023 16:27:53 1538236 22293843 oifs_43r3_ps_0252_2000050100_123_969_12185896_1 864,000 5,908 0.0068


©2024 climateprediction.net