climateprediction.net home page
Task 22280498

Task 22280498

Name oifs_43r3_ps_0995_2008050100_123_977_12194639_0
Workunit 12194639
Created 23 Dec 2022, 19:53:25 UTC
Sent 23 Dec 2022, 20:05:58 UTC
Report deadline 22 Jan 2023, 20:05:58 UTC
Received 1 Jan 2023, 22:55:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1520317
Run time 20 hours 23 min 35 sec
CPU time 18 hours 52 min 51 sec
Validate state Invalid
Credit 12,700.80
Device peak FLOPS 8.55 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.63 MB
Peak swap size 4,978.16 MB
Peak disk usage 7,377.58 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>

  04:08:15 STEP 2141 H=2141:00 +CPU= 15.428
  04:08:41 STEP 2142 H=2142:00 +CPU= 24.816
  04:08:57 STEP 2143 H=2143:00 +CPU= 15.632
  04:09:14 STEP 2144 H=2144:00 +CPU= 16.077
  04:09:39 STEP 2145 H=2145:00 +CPU= 24.148
  04:09:55 STEP 2146 H=2146:00 +CPU= 15.947
  04:10:11 STEP 2147 H=2147:00 +CPU= 15.330
  04:10:41 STEP 2148 H=2148:00 +CPU= 28.076
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002148
  04:10:59 STEP 2149 H=2149:00 +CPU= 17.082
  04:11:17 STEP 2150 H=2150:00 +CPU= 17.632
  04:11:44 STEP 2151 H=2151:00 +CPU= 25.357
  04:12:00 STEP 2152 H=2152:00 +CPU= 15.509
  04:12:14 STEP 2153 H=2153:00 +CPU= 14.249
  04:12:39 STEP 2154 H=2154:00 +CPU= 24.914
  04:12:56 STEP 2155 H=2155:00 +CPU= 16.500
  04:13:13 STEP 2156 H=2156:00 +CPU= 17.098
  04:13:43 STEP 2157 H=2157:00 +CPU= 28.409
  04:14:02 STEP 2158 H=2158:00 +CPU= 18.579
  04:14:21 STEP 2159 H=2159:00 +CPU= 18.548
  04:14:56 STEP 2160 H=2160:00 +CPU= 33.410
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_89.zip
Uploading the intermediate file: upload_file_89.zip
  04:15:13 STEP 2161 H=2161:00 +CPU= 16.974
  04:15:32 STEP 2162 H=2162:00 +CPU= 17.924
  04:16:01 STEP 2163 H=2163:00 +CPU= 27.894
  04:16:20 STEP 2164 H=2164:00 +CPU= 17.915
  04:16:38 STEP 2165 H=2165:00 +CPU= 17.845
  04:17:07 STEP 2166 H=2166:00 +CPU= 27.652
  04:17:24 STEP 2167 H=2167:00 +CPU= 17.118
  04:17:42 STEP 2168 H=2168:00 +CPU= 17.554
  04:18:10 STEP 2169 H=2169:00 +CPU= 26.961
  04:18:29 STEP 2170 H=2170:00 +CPU= 18.270
  04:18:48 STEP 2171 H=2171:00 +CPU= 17.878
  04:19:20 STEP 2172 H=2172:00 +CPU= 30.972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002172
  04:19:38 STEP 2173 H=2173:00 +CPU= 17.334
  04:19:56 STEP 2174 H=2174:00 +CPU= 17.487
  04:20:25 STEP 2175 H=2175:00 +CPU= 27.882
  04:20:45 STEP 2176 H=2176:00 +CPU= 18.125
  04:21:04 STEP 2177 H=2177:00 +CPU= 17.801
  04:21:30 STEP 2178 H=2178:00 +CPU= 26.446
  04:21:50 STEP 2179 H=2179:00 +CPU= 19.082
  04:22:10 STEP 2180 H=2180:00 +CPU= 19.215
  04:22:42 STEP 2181 H=2181:00 +CPU= 29.416
  04:23:04 STEP 2182 H=2182:00 +CPU= 19.843
  04:23:25 STEP 2183 H=2183:00 +CPU= 19.813
  04:24:01 STEP 2184 H=2184:00 +CPU= 34.277
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_90.zip
Uploading the intermediate file: upload_file_90.zip
  04:24:20 STEP 2185 H=2185:00 +CPU= 18.657
  04:24:40 STEP 2186 H=2186:00 +CPU= 19.352
  04:25:08 STEP 2187 H=2187:00 +CPU= 27.397
  04:25:26 STEP 2188 H=2188:00 +CPU= 17.268
  04:25:43 STEP 2189 H=2189:00 +CPU= 17.101
  04:26:11 STEP 2190 H=2190:00 +CPU= 27.561
  04:26:27 STEP 2191 H=2191:00 +CPU= 15.631
  04:26:43 STEP 2192 H=2192:00 +CPU= 15.492
  04:27:06 STEP 2193 H=2193:00 +CPU= 23.644
  04:27:23 STEP 2194 H=2194:00 +CPU= 16.815
  04:27:43 STEP 2195 H=2195:00 +CPU= 18.730
  04:28:16 STEP 2196 H=2196:00 +CPU= 31.638
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002196
  04:28:35 STEP 2197 H=2197:00 +CPU= 18.451
  04:28:55 STEP 2198 H=2198:00 +CPU= 18.861
  04:29:26 STEP 2199 H=2199:00 +CPU= 28.408
  04:29:46 STEP 2200 H=2200:00 +CPU= 18.843
  04:30:06 STEP 2201 H=2201:00 +CPU= 19.156
  04:30:35 STEP 2202 H=2202:00 +CPU= 29.352
  04:30:54 STEP 2203 H=2203:00 +CPU= 18.519
  04:31:14 STEP 2204 H=2204:00 +CPU= 18.617
  04:31:41 STEP 2205 H=2205:00 +CPU= 27.024
  04:32:00 STEP 2206 H=2206:00 +CPU= 18.029
  04:32:19 STEP 2207 H=2207:00 +CPU= 18.796
  04:32:56 STEP 2208 H=2208:00 +CPU= 34.062
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_91.zip
Uploading the intermediate file: upload_file_91.zip
  04:33:13 STEP 2209 H=2209:00 +CPU= 17.043
  04:33:30 STEP 2210 H=2210:00 +CPU= 17.190
  04:33:57 STEP 2211 H=2211:00 +CPU= 26.625
  04:34:15 STEP 2212 H=2212:00 +CPU= 17.302
  04:34:31 STEP 2213 H=2213:00 +CPU= 16.576
  04:34:59 STEP 2214 H=2214:00 +CPU= 27.647
  04:35:17 STEP 2215 H=2215:00 +CPU= 17.470
  04:35:36 STEP 2216 H=2216:00 +CPU= 18.204
  04:36:04 STEP 2217 H=2217:00 +CPU= 27.642
  04:36:25 STEP 2218 H=2218:00 +CPU= 19.730
  04:36:44 STEP 2219 H=2219:00 +CPU= 18.443
  04:37:18 STEP 2220 H=2220:00 +CPU= 32.188
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002220
  04:37:38 STEP 2221 H=2221:00 +CPU= 18.743
  04:37:59 STEP 2222 H=2222:00 +CPU= 19.885
  04:38:27 STEP 2223 H=2223:00 +CPU= 27.007
  04:38:45 STEP 2224 H=2224:00 +CPU= 17.201
  04:39:03 STEP 2225 H=2225:00 +CPU= 17.126
  04:39:31 STEP 2226 H=2226:00 +CPU= 26.757
  04:39:49 STEP 2227 H=2227:00 +CPU= 16.920
  04:40:07 STEP 2228 H=2228:00 +CPU= 17.771
  04:40:35 STEP 2229 H=2229:00 +CPU= 26.783
  04:40:54 STEP 2230 H=2230:00 +CPU= 18.063
  04:41:13 STEP 2231 H=2231:00 +CPU= 18.277
  04:41:46 STEP 2232 H=2232:00 +CPU= 31.858
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_92.zip
Uploading the intermediate file: upload_file_92.zip
  04:42:04 STEP 2233 H=2233:00 +CPU= 16.793
  04:42:22 STEP 2234 H=2234:00 +CPU= 17.047
  04:42:49 STEP 2235 H=2235:00 +CPU= 26.169
  04:43:07 STEP 2236 H=2236:00 +CPU= 17.695
  04:43:25 STEP 2237 H=2237:00 +CPU= 16.757
  04:43:55 STEP 2238 H=2238:00 +CPU= 29.536
  04:44:13 STEP 2239 H=2239:00 +CPU= 17.647
  04:44:33 STEP 2240 H=2240:00 +CPU= 19.124
  04:45:03 STEP 2241 H=2241:00 +CPU= 28.881
  04:45:22 STEP 2242 H=2242:00 +CPU= 19.340
  04:45:41 STEP 2243 H=2243:00 +CPU= 18.853
  04:46:17 STEP 2244 H=2244:00 +CPU= 34.419
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002244
  04:46:39 STEP 2245 H=2245:00 +CPU= 20.207
  04:47:02 STEP 2246 H=2246:00 +CPU= 21.731
  04:47:35 STEP 2247 H=2247:00 +CPU= 31.361
  04:47:57 STEP 2248 H=2248:00 +CPU= 20.592
  04:48:19 STEP 2249 H=2249:00 +CPU= 21.104
  04:48:52 STEP 2250 H=2250:00 +CPU= 30.938
  04:49:12 STEP 2251 H=2251:00 +CPU= 19.437
  04:49:31 STEP 2252 H=2252:00 +CPU= 18.926
  04:50:01 STEP 2253 H=2253:00 +CPU= 29.401
  04:50:21 STEP 2254 H=2254:00 +CPU= 19.769
  04:50:42 STEP 2255 H=2255:00 +CPU= 20.082
  04:51:22 STEP 2256 H=2256:00 +CPU= 37.638
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_93.zip
Uploading the intermediate file: upload_file_93.zip
  04:51:42 STEP 2257 H=2257:00 +CPU= 18.947
  04:52:01 STEP 2258 H=2258:00 +CPU= 19.070
  04:52:30 STEP 2259 H=2259:00 +CPU= 29.160
  04:52:51 STEP 2260 H=2260:00 +CPU= 20.254
  04:53:12 STEP 2261 H=2261:00 +CPU= 20.026
  04:53:43 STEP 2262 H=2262:00 +CPU= 30.394
  04:54:01 STEP 2263 H=2263:00 +CPU= 18.458
  04:54:20 STEP 2264 H=2264:00 +CPU= 18.470
  04:54:50 STEP 2265 H=2265:00 +CPU= 29.028
  04:55:08 STEP 2266 H=2266:00 +CPU= 18.349
  04:55:29 STEP 2267 H=2267:00 +CPU= 20.355
  04:56:03 STEP 2268 H=2268:00 +CPU= 33.936
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002268
  04:56:28 STEP 2269 H=2269:00 +CPU= 23.945
  04:56:55 STEP 2270 H=2270:00 +CPU= 26.198
  04:57:32 STEP 2271 H=2271:00 +CPU= 36.089
  04:57:59 STEP 2272 H=2272:00 +CPU= 26.538
  04:58:27 STEP 2273 H=2273:00 +CPU= 27.018
  04:59:15 STEP 2274 H=2274:00 +CPU= 44.487
  04:59:45 STEP 2275 H=2275:00 +CPU= 27.704
  05:00:17 STEP 2276 H=2276:00 +CPU= 30.888
  05:01:02 STEP 2277 H=2277:00 +CPU= 42.251
  05:01:34 STEP 2278 H=2278:00 +CPU= 29.966
  05:02:07 STEP 2279 H=2279:00 +CPU= 29.371
  05:02:56 STEP 2280 H=2280:00 +CPU= 47.824
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_94.zip
Uploading the intermediate file: upload_file_94.zip
  05:03:27 STEP 2281 H=2281:00 +CPU= 27.536
  05:04:04 STEP 2282 H=2282:00 +CPU= 32.858
  05:04:53 STEP 2283 H=2283:00 +CPU= 44.270
  05:05:24 STEP 2284 H=2284:00 +CPU= 28.739
  05:05:53 STEP 2285 H=2285:00 +CPU= 25.619
  05:06:44 STEP 2286 H=2286:00 +CPU= 49.020
  05:07:15 STEP 2287 H=2287:00 +CPU= 30.397
  05:07:45 STEP 2288 H=2288:00 +CPU= 28.439
  05:08:31 STEP 2289 H=2289:00 +CPU= 43.990
  05:09:04 STEP 2290 H=2290:00 +CPU= 30.970
  05:09:33 STEP 2291 H=2291:00 +CPU= 27.778
  05:10:21 STEP 2292 H=2292:00 +CPU= 44.601
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002292
  05:10:51 STEP 2293 H=2293:00 +CPU= 28.025
  05:11:23 STEP 2294 H=2294:00 +CPU= 29.698
  05:12:03 STEP 2295 H=2295:00 +CPU= 37.360
  05:12:32 STEP 2296 H=2296:00 +CPU= 27.480
  05:13:00 STEP 2297 H=2297:00 +CPU= 26.649
  05:13:44 STEP 2298 H=2298:00 +CPU= 41.097
  05:14:15 STEP 2299 H=2299:00 +CPU= 28.736
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0995
(argv4) batchid: 977
(argv5) wuid: 12194639
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c90e9624dbe275fad1b4239e3428bd64 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0995_2008050100_123_977_12194639.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0995_2008050100_123_977_12194639.zip
ic_ancil_file: ic_ancil_12194639
ifsdata_file: ifsdata_12194639
climate_data_file: clim_data_12194639
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2c2a7a3ea0e011560d64b9bd9de65616 to: /var/lib/boinc-client/slots/4/ic_ancil_12194639.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12194639.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8bb064d0242eade7f5ca6bdfc7caf47e to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12194639.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12194639.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12194639.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12194639.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12194639.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12194639.xml
-- Model is restarting --
Adjusting last_iter, 2300, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12194639.xml
last_cpu_time: 55057
upload_file_number: 95
last_iter: 2279
last_upload: 8208000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869
The child process has been launched with process id: 59989
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:roo:1:1:59989:59989] [20230102:051525:1672596925:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:16:17 STEP 2280 H=2280:00 +CPU= 44.892
  05:16:42 STEP 2281 H=2281:00 +CPU= 23.625
  05:17:11 STEP 2282 H=2282:00 +CPU= 27.472
  05:17:59 STEP 2283 H=2283:00 +CPU= 44.881
  05:18:32 STEP 2284 H=2284:00 +CPU= 30.875
  05:19:01 STEP 2285 H=2285:00 +CPU= 26.865
  05:19:50 STEP 2286 H=2286:00 +CPU= 45.201
  05:20:24 STEP 2287 H=2287:00 +CPU= 30.642
  05:20:53 STEP 2288 H=2288:00 +CPU= 28.310
  05:21:39 STEP 2289 H=2289:00 +CPU= 42.770
  05:22:09 STEP 2290 H=2290:00 +CPU= 28.838
  05:22:37 STEP 2291 H=2291:00 +CPU= 27.416
  05:23:24 STEP 2292 H=2292:00 +CPU= 46.100
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002292
  05:23:52 STEP 2293 H=2293:00 +CPU= 27.294
  05:24:18 STEP 2294 H=2294:00 +CPU= 24.806
  05:25:06 STEP 2295 H=2295:00 +CPU= 44.495
  05:25:34 STEP 2296 H=2296:00 +CPU= 27.229
  05:26:02 STEP 2297 H=2297:00 +CPU= 26.275
  05:26:45 STEP 2298 H=2298:00 +CPU= 41.082
  05:27:14 STEP 2299 H=2299:00 +CPU= 27.105
  05:27:42 STEP 2300 H=2300:00 +CPU= 26.300
  05:28:26 STEP 2301 H=2301:00 +CPU= 41.946
  05:28:56 STEP 2302 H=2302:00 +CPU= 28.003
  05:29:26 STEP 2303 H=2303:00 +CPU= 28.334
  05:30:14 STEP 2304 H=2304:00 +CPU= 45.113
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_95.zip
Uploading the intermediate file: upload_file_95.zip
  05:30:41 STEP 2305 H=2305:00 +CPU= 25.248
  05:31:07 STEP 2306 H=2306:00 +CPU= 24.655
  05:31:46 STEP 2307 H=2307:00 +CPU= 37.810
  05:32:11 STEP 2308 H=2308:00 +CPU= 23.902
  05:32:36 STEP 2309 H=2309:00 +CPU= 23.849
  05:33:17 STEP 2310 H=2310:00 +CPU= 39.687
  05:33:41 STEP 2311 H=2311:00 +CPU= 22.409
  05:34:06 STEP 2312 H=2312:00 +CPU= 23.972
  05:34:46 STEP 2313 H=2313:00 +CPU= 37.410
  05:35:14 STEP 2314 H=2314:00 +CPU= 26.227
  05:35:38 STEP 2315 H=2315:00 +CPU= 23.103
  05:36:29 STEP 2316 H=2316:00 +CPU= 47.690
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002316
  05:36:56 STEP 2317 H=2317:00 +CPU= 25.275
  05:37:26 STEP 2318 H=2318:00 +CPU= 27.600
  05:38:07 STEP 2319 H=2319:00 +CPU= 37.885
  05:38:34 STEP 2320 H=2320:00 +CPU= 25.232
  05:38:59 STEP 2321 H=2321:00 +CPU= 23.265
  05:39:36 STEP 2322 H=2322:00 +CPU= 34.713
  05:39:59 STEP 2323 H=2323:00 +CPU= 21.763
  05:40:23 STEP 2324 H=2324:00 +CPU= 22.624
  05:41:00 STEP 2325 H=2325:00 +CPU= 34.864
  05:41:23 STEP 2326 H=2326:00 +CPU= 22.322
  05:41:47 STEP 2327 H=2327:00 +CPU= 22.487
  05:42:30 STEP 2328 H=2328:00 +CPU= 41.178
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_96.zip
Uploading the intermediate file: upload_file_96.zip
  05:42:54 STEP 2329 H=2329:00 +CPU= 22.373
  05:43:17 STEP 2330 H=2330:00 +CPU= 22.264
  05:43:53 STEP 2331 H=2331:00 +CPU= 34.257
  05:44:16 STEP 2332 H=2332:00 +CPU= 22.408
  05:44:39 STEP 2333 H=2333:00 +CPU= 21.515
  05:45:17 STEP 2334 H=2334:00 +CPU= 35.348
  05:45:40 STEP 2335 H=2335:00 +CPU= 22.094
  05:46:03 STEP 2336 H=2336:00 +CPU= 22.726
  05:46:39 STEP 2337 H=2337:00 +CPU= 34.892
  05:47:03 STEP 2338 H=2338:00 +CPU= 22.595
  05:47:27 STEP 2339 H=2339:00 +CPU= 22.533
  05:48:08 STEP 2340 H=2340:00 +CPU= 38.889
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002340
  05:48:31 STEP 2341 H=2341:00 +CPU= 22.111
  05:48:56 STEP 2342 H=2342:00 +CPU= 23.158
  05:49:35 STEP 2343 H=2343:00 +CPU= 36.527
  05:50:00 STEP 2344 H=2344:00 +CPU= 23.969
  05:50:25 STEP 2345 H=2345:00 +CPU= 22.734
  05:51:03 STEP 2346 H=2346:00 +CPU= 35.287
  05:51:28 STEP 2347 H=2347:00 +CPU= 22.895
  05:51:51 STEP 2348 H=2348:00 +CPU= 22.993
  05:52:28 STEP 2349 H=2349:00 +CPU= 35.124
  05:52:53 STEP 2350 H=2350:00 +CPU= 23.429
  05:53:17 STEP 2351 H=2351:00 +CPU= 22.627
  05:54:01 STEP 2352 H=2352:00 +CPU= 42.422
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_97.zip
Uploading the intermediate file: upload_file_97.zip
  05:54:27 STEP 2353 H=2353:00 +CPU= 23.401
  05:54:52 STEP 2354 H=2354:00 +CPU= 24.052
  05:55:28 STEP 2355 H=2355:00 +CPU= 34.776
  05:55:51 STEP 2356 H=2356:00 +CPU= 22.412
  05:56:15 STEP 2357 H=2357:00 +CPU= 23.123
  05:56:53 STEP 2358 H=2358:00 +CPU= 35.537
  05:57:17 STEP 2359 H=2359:00 +CPU= 22.707
  05:57:41 STEP 2360 H=2360:00 +CPU= 22.975
  05:58:17 STEP 2361 H=2361:00 +CPU= 34.122
  05:58:41 STEP 2362 H=2362:00 +CPU= 23.527
  05:59:04 STEP 2363 H=2363:00 +CPU= 21.844
  05:59:43 STEP 2364 H=2364:00 +CPU= 37.495
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002364
  06:00:07 STEP 2365 H=2365:00 +CPU= 22.151
  06:00:31 STEP 2366 H=2366:00 +CPU= 22.810
  06:01:06 STEP 2367 H=2367:00 +CPU= 33.539
  06:01:31 STEP 2368 H=2368:00 +CPU= 23.659
  06:01:55 STEP 2369 H=2369:00 +CPU= 22.747
  06:02:34 STEP 2370 H=2370:00 +CPU= 36.456
  06:02:57 STEP 2371 H=2371:00 +CPU= 22.252
  06:03:21 STEP 2372 H=2372:00 +CPU= 22.622
  06:03:58 STEP 2373 H=2373:00 +CPU= 35.045
  06:04:23 STEP 2374 H=2374:00 +CPU= 22.718
  06:04:47 STEP 2375 H=2375:00 +CPU= 23.186
  06:05:31 STEP 2376 H=2376:00 +CPU= 41.147
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_98.zip
Uploading the intermediate file: upload_file_98.zip
  06:05:58 STEP 2377 H=2377:00 +CPU= 25.540
  06:06:24 STEP 2378 H=2378:00 +CPU= 25.439
  06:07:04 STEP 2379 H=2379:00 +CPU= 36.961
  06:07:29 STEP 2380 H=2380:00 +CPU= 23.909
  06:07:55 STEP 2381 H=2381:00 +CPU= 25.164
  06:08:37 STEP 2382 H=2382:00 +CPU= 39.834
  06:09:03 STEP 2383 H=2383:00 +CPU= 24.726
  06:09:29 STEP 2384 H=2384:00 +CPU= 25.735
  06:10:09 STEP 2385 H=2385:00 +CPU= 37.621
  06:10:35 STEP 2386 H=2386:00 +CPU= 25.340
  06:11:02 STEP 2387 H=2387:00 +CPU= 24.560
  06:11:45 STEP 2388 H=2388:00 +CPU= 41.110
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002388
  06:12:12 STEP 2389 H=2389:00 +CPU= 25.176
  06:12:40 STEP 2390 H=2390:00 +CPU= 26.110
  06:13:20 STEP 2391 H=2391:00 +CPU= 38.167
  06:13:46 STEP 2392 H=2392:00 +CPU= 25.122
  06:14:12 STEP 2393 H=2393:00 +CPU= 24.538
  06:14:54 STEP 2394 H=2394:00 +CPU= 40.491
  06:15:25 STEP 2395 H=2395:00 +CPU= 29.034
  06:15:55 STEP 2396 H=2396:00 +CPU= 29.330
  06:16:45 STEP 2397 H=2397:00 +CPU= 47.105
  06:17:22 STEP 2398 H=2398:00 +CPU= 34.846
  06:17:59 STEP 2399 H=2399:00 +CPU= 34.409
  06:19:01 STEP 2400 H=2400:00 +CPU= 58.466
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_99.zip
Uploading the intermediate file: upload_file_99.zip
  06:19:35 STEP 2401 H=2401:00 +CPU= 33.129
  06:20:12 STEP 2402 H=2402:00 +CPU= 34.180
  06:21:09 STEP 2403 H=2403:00 +CPU= 53.156
  06:21:49 STEP 2404 H=2404:00 +CPU= 37.116
  06:22:29 STEP 2405 H=2405:00 +CPU= 36.875
  06:23:30 STEP 2406 H=2406:00 +CPU= 54.831
  06:24:10 STEP 2407 H=2407:00 +CPU= 36.228
  06:24:48 STEP 2408 H=2408:00 +CPU= 36.340
  06:25:47 STEP 2409 H=2409:00 +CPU= 53.944
  06:26:26 STEP 2410 H=2410:00 +CPU= 36.282
  06:27:05 STEP 2411 H=2411:00 +CPU= 36.114
  06:28:08 STEP 2412 H=2412:00 +CPU= 59.474
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002412
  06:28:48 STEP 2413 H=2413:00 +CPU= 35.813
  06:29:28 STEP 2414 H=2414:00 +CPU= 37.146
  06:30:25 STEP 2415 H=2415:00 +CPU= 53.291
  06:31:04 STEP 2416 H=2416:00 +CPU= 36.110
  06:31:42 STEP 2417 H=2417:00 +CPU= 35.537
  06:32:38 STEP 2418 H=2418:00 +CPU= 52.538
  06:33:15 STEP 2419 H=2419:00 +CPU= 33.986
  06:33:54 STEP 2420 H=2420:00 +CPU= 36.460
  06:34:52 STEP 2421 H=2421:00 +CPU= 53.215
  06:35:33 STEP 2422 H=2422:00 +CPU= 37.861
  06:36:12 STEP 2423 H=2423:00 +CPU= 35.694
  06:37:18 STEP 2424 H=2424:00 +CPU= 61.816
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_100.zip
Uploading the intermediate file: upload_file_100.zip
  06:37:55 STEP 2425 H=2425:00 +CPU= 34.544
  06:38:35 STEP 2426 H=2426:00 +CPU= 36.306
  06:39:29 STEP 2427 H=2427:00 +CPU= 50.454
  06:40:06 STEP 2428 H=2428:00 +CPU= 34.013
  06:40:43 STEP 2429 H=2429:00 +CPU= 34.811
  06:41:43 STEP 2430 H=2430:00 +CPU= 55.386
  06:42:26 STEP 2431 H=2431:00 +CPU= 38.279
  06:43:05 STEP 2432 H=2432:00 +CPU= 36.115
  06:44:02 STEP 2433 H=2433:00 +CPU= 51.493
  06:44:41 STEP 2434 H=2434:00 +CPU= 35.030
  06:45:20 STEP 2435 H=2435:00 +CPU= 36.751
  06:46:24 STEP 2436 H=2436:00 +CPU= 58.285
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002436
  06:47:04 STEP 2437 H=2437:00 +CPU= 36.729
  06:47:44 STEP 2438 H=2438:00 +CPU= 36.749
  06:48:47 STEP 2439 H=2439:00 +CPU= 57.610
  06:49:29 STEP 2440 H=2440:00 +CPU= 38.241
  06:50:06 STEP 2441 H=2441:00 +CPU= 33.105
  06:51:08 STEP 2442 H=2442:00 +CPU= 56.144
  06:51:47 STEP 2443 H=2443:00 +CPU= 36.385
  06:52:26 STEP 2444 H=2444:00 +CPU= 35.519
  06:53:22 STEP 2445 H=2445:00 +CPU= 52.492
  06:54:01 STEP 2446 H=2446:00 +CPU= 35.637
  06:54:41 STEP 2447 H=2447:00 +CPU= 37.416
  06:55:47 STEP 2448 H=2448:00 +CPU= 61.585
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_101.zip
Uploading the intermediate file: upload_file_101.zip
  06:56:25 STEP 2449 H=2449:00 +CPU= 36.071
  06:57:03 STEP 2450 H=2450:00 +CPU= 35.328
  06:57:56 STEP 2451 H=2451:00 +CPU= 50.009
  06:58:36 STEP 2452 H=2452:00 +CPU= 36.669
  06:59:12 STEP 2453 H=2453:00 +CPU= 34.252
  07:00:08 STEP 2454 H=2454:00 +CPU= 51.921
  07:00:43 STEP 2455 H=2455:00 +CPU= 32.355
  07:01:11 STEP 2456 H=2456:00 +CPU= 25.946
  07:01:50 STEP 2457 H=2457:00 +CPU= 36.694
  07:02:16 STEP 2458 H=2458:00 +CPU= 24.863
  07:02:44 STEP 2459 H=2459:00 +CPU= 25.178
  07:03:30 STEP 2460 H=2460:00 +CPU= 42.104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002460
  07:03:56 STEP 2461 H=2461:00 +CPU= 23.845
  07:04:22 STEP 2462 H=2462:00 +CPU= 24.903
  07:05:04 STEP 2463 H=2463:00 +CPU= 37.913
  07:05:32 STEP 2464 H=2464:00 +CPU= 26.112
  07:05:57 STEP 2465 H=2465:00 +CPU= 23.514
  07:06:38 STEP 2466 H=2466:00 +CPU= 37.972
  07:07:05 STEP 2467 H=2467:00 +CPU= 25.114
  07:07:31 STEP 2468 H=2468:00 +CPU= 24.233
  07:08:08 STEP 2469 H=2469:00 +CPU= 35.374
  07:08:34 STEP 2470 H=2470:00 +CPU= 24.037
  07:08:59 STEP 2471 H=2471:00 +CPU= 23.552
  07:09:43 STEP 2472 H=2472:00 +CPU= 41.785
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_102.zip
Uploading the intermediate file: upload_file_102.zip
  07:10:06 STEP 2473 H=2473:00 +CPU= 21.889
  07:10:30 STEP 2474 H=2474:00 +CPU= 22.904
  07:11:06 STEP 2475 H=2475:00 +CPU= 34.269
  07:11:31 STEP 2476 H=2476:00 +CPU= 23.463
  07:11:54 STEP 2477 H=2477:00 +CPU= 22.284
  07:12:31 STEP 2478 H=2478:00 +CPU= 35.378
  07:12:55 STEP 2479 H=2479:00 +CPU= 22.788
  07:13:19 STEP 2480 H=2480:00 +CPU= 22.538
  07:13:53 STEP 2481 H=2481:00 +CPU= 32.926
  07:14:17 STEP 2482 H=2482:00 +CPU= 22.325
  07:14:37 STEP 2483 H=2483:00 +CPU= 20.048
  07:15:16 STEP 2484 H=2484:00 +CPU= 36.697
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002484
  07:15:39 STEP 2485 H=2485:00 +CPU= 21.297
  07:16:03 STEP 2486 H=2486:00 +CPU= 22.180
  07:16:41 STEP 2487 H=2487:00 +CPU= 34.722
  07:17:06 STEP 2488 H=2488:00 +CPU= 22.842
  07:17:29 STEP 2489 H=2489:00 +CPU= 22.192
  07:18:05 STEP 2490 H=2490:00 +CPU= 34.065
  07:18:27 STEP 2491 H=2491:00 +CPU= 21.468
  07:18:50 STEP 2492 H=2492:00 +CPU= 22.095
  07:19:25 STEP 2493 H=2493:00 +CPU= 33.523
  07:19:50 STEP 2494 H=2494:00 +CPU= 22.391
  07:20:14 STEP 2495 H=2495:00 +CPU= 22.394
  07:20:55 STEP 2496 H=2496:00 +CPU= 39.347
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_103.zip
Uploading the intermediate file: upload_file_103.zip
  07:21:18 STEP 2497 H=2497:00 +CPU= 21.187
  07:21:41 STEP 2498 H=2498:00 +CPU= 21.954
  07:22:16 STEP 2499 H=2499:00 +CPU= 32.667
  07:22:39 STEP 2500 H=2500:00 +CPU= 22.295
  07:23:01 STEP 2501 H=2501:00 +CPU= 21.126
  07:23:36 STEP 2502 H=2502:00 +CPU= 33.331
  07:23:59 STEP 2503 H=2503:00 +CPU= 21.710
  07:24:23 STEP 2504 H=2504:00 +CPU= 22.603
  07:24:58 STEP 2505 H=2505:00 +CPU= 33.000
  07:25:21 STEP 2506 H=2506:00 +CPU= 22.022
  07:25:45 STEP 2507 H=2507:00 +CPU= 21.776
  07:26:23 STEP 2508 H=2508:00 +CPU= 36.876
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002508
  07:26:48 STEP 2509 H=2509:00 +CPU= 22.533
  07:27:11 STEP 2510 H=2510:00 +CPU= 22.076
  07:27:48 STEP 2511 H=2511:00 +CPU= 34.497
  07:28:13 STEP 2512 H=2512:00 +CPU= 22.260
  07:28:38 STEP 2513 H=2513:00 +CPU= 22.644
  07:29:18 STEP 2514 H=2514:00 +CPU= 35.710
  07:29:42 STEP 2515 H=2515:00 +CPU= 23.188
  07:30:07 STEP 2516 H=2516:00 +CPU= 23.292
  07:30:43 STEP 2517 H=2517:00 +CPU= 34.407
  07:31:08 STEP 2518 H=2518:00 +CPU= 23.966
  07:31:31 STEP 2519 H=2519:00 +CPU= 22.681
  07:32:12 STEP 2520 H=2520:00 +CPU= 39.992
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_104.zip
Uploading the intermediate file: upload_file_104.zip
  07:32:35 STEP 2521 H=2521:00 +CPU= 21.856
Uploading trickle at timestep: 9072000
  07:32:59 STEP 2522 H=2522:00 +CPU= 22.427
  07:33:35 STEP 2523 H=2523:00 +CPU= 33.775
  07:33:58 STEP 2524 H=2524:00 +CPU= 22.291
  07:34:22 STEP 2525 H=2525:00 +CPU= 21.715
  07:34:58 STEP 2526 H=2526:00 +CPU= 34.466
  07:35:22 STEP 2527 H=2527:00 +CPU= 22.474
  07:35:47 STEP 2528 H=2528:00 +CPU= 23.542
  07:36:22 STEP 2529 H=2529:00 +CPU= 34.220
  07:36:47 STEP 2530 H=2530:00 +CPU= 23.377
  07:37:12 STEP 2531 H=2531:00 +CPU= 23.292
  07:37:53 STEP 2532 H=2532:00 +CPU= 38.954
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002532
  07:38:17 STEP 2533 H=2533:00 +CPU= 22.621
  07:38:41 STEP 2534 H=2534:00 +CPU= 22.858
  07:39:18 STEP 2535 H=2535:00 +CPU= 34.527
  07:39:43 STEP 2536 H=2536:00 +CPU= 23.477
  07:40:06 STEP 2537 H=2537:00 +CPU= 22.206
  07:40:44 STEP 2538 H=2538:00 +CPU= 35.783
  07:41:09 STEP 2539 H=2539:00 +CPU= 23.213
  07:41:37 STEP 2540 H=2540:00 +CPU= 24.262
  07:42:17 STEP 2541 H=2541:00 +CPU= 35.987
  07:42:40 STEP 2542 H=2542:00 +CPU= 22.223
  07:43:05 STEP 2543 H=2543:00 +CPU= 23.275
  07:43:46 STEP 2544 H=2544:00 +CPU= 40.169
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_105.zip
Uploading the intermediate file: upload_file_105.zip
  07:44:11 STEP 2545 H=2545:00 +CPU= 23.093
  07:44:36 STEP 2546 H=2546:00 +CPU= 23.212
  07:45:14 STEP 2547 H=2547:00 +CPU= 35.194
  07:45:40 STEP 2548 H=2548:00 +CPU= 24.063
  07:46:05 STEP 2549 H=2549:00 +CPU= 23.812
  07:46:46 STEP 2550 H=2550:00 +CPU= 37.052
  07:47:12 STEP 2551 H=2551:00 +CPU= 23.126
  07:47:39 STEP 2552 H=2552:00 +CPU= 23.562
  07:48:16 STEP 2553 H=2553:00 +CPU= 34.822
  07:48:39 STEP 2554 H=2554:00 +CPU= 22.886
  07:49:03 STEP 2555 H=2555:00 +CPU= 22.495
  07:49:44 STEP 2556 H=2556:00 +CPU= 38.989
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002556
  07:50:08 STEP 2557 H=2557:00 +CPU= 22.663
  07:50:32 STEP 2558 H=2558:00 +CPU= 23.114
  07:51:10 STEP 2559 H=2559:00 +CPU= 35.410
  07:51:35 STEP 2560 H=2560:00 +CPU= 23.268
  07:51:58 STEP 2561 H=2561:00 +CPU= 21.620
  07:52:35 STEP 2562 H=2562:00 +CPU= 35.188
  07:52:57 STEP 2563 H=2563:00 +CPU= 22.013
  07:53:22 STEP 2564 H=2564:00 +CPU= 23.202
  07:53:59 STEP 2565 H=2565:00 +CPU= 34.849
  07:54:24 STEP 2566 H=2566:00 +CPU= 23.050
  07:54:49 STEP 2567 H=2567:00 +CPU= 23.255
  07:55:32 STEP 2568 H=2568:00 +CPU= 40.561
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_106.zip
Uploading the intermediate file: upload_file_106.zip
  07:55:55 STEP 2569 H=2569:00 +CPU= 21.645
  07:56:18 STEP 2570 H=2570:00 +CPU= 21.647
  07:56:53 STEP 2571 H=2571:00 +CPU= 34.236
  07:57:20 STEP 2572 H=2572:00 +CPU= 25.482
  07:57:54 STEP 2573 H=2573:00 +CPU= 29.621
  07:58:45 STEP 2574 H=2574:00 +CPU= 44.651
  07:59:18 STEP 2575 H=2575:00 +CPU= 28.276
  07:59:47 STEP 2576 H=2576:00 +CPU= 27.928
  08:00:26 STEP 2577 H=2577:00 +CPU= 37.843
  08:00:55 STEP 2578 H=2578:00 +CPU= 27.422
  08:01:26 STEP 2579 H=2579:00 +CPU= 28.323
  08:02:15 STEP 2580 H=2580:00 +CPU= 45.121
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002580
  08:02:43 STEP 2581 H=2581:00 +CPU= 26.923
  08:03:11 STEP 2582 H=2582:00 +CPU= 27.142
  08:03:56 STEP 2583 H=2583:00 +CPU= 40.941
  08:04:24 STEP 2584 H=2584:00 +CPU= 26.700
  08:04:53 STEP 2585 H=2585:00 +CPU= 27.344
  08:05:37 STEP 2586 H=2586:00 +CPU= 41.060
  08:06:06 STEP 2587 H=2587:00 +CPU= 26.733
  08:06:36 STEP 2588 H=2588:00 +CPU= 28.441
  08:07:19 STEP 2589 H=2589:00 +CPU= 41.131
  08:07:50 STEP 2590 H=2590:00 +CPU= 28.121
  08:08:21 STEP 2591 H=2591:00 +CPU= 28.776
  08:09:11 STEP 2592 H=2592:00 +CPU= 47.600
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_107.zip
Uploading the intermediate file: upload_file_107.zip
  08:09:39 STEP 2593 H=2593:00 +CPU= 26.326
  08:10:09 STEP 2594 H=2594:00 +CPU= 27.866
  08:10:54 STEP 2595 H=2595:00 +CPU= 41.083
  08:11:22 STEP 2596 H=2596:00 +CPU= 26.438
  08:11:47 STEP 2597 H=2597:00 +CPU= 25.376
  08:12:33 STEP 2598 H=2598:00 +CPU= 41.262
  08:13:00 STEP 2599 H=2599:00 +CPU= 26.955
  08:13:31 STEP 2600 H=2600:00 +CPU= 28.439
  08:14:17 STEP 2601 H=2601:00 +CPU= 40.526
  08:14:48 STEP 2602 H=2602:00 +CPU= 27.155
  08:15:19 STEP 2603 H=2603:00 +CPU= 26.899
  08:16:11 STEP 2604 H=2604:00 +CPU= 46.626
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002604
  08:16:40 STEP 2605 H=2605:00 +CPU= 27.666
  08:17:11 STEP 2606 H=2606:00 +CPU= 27.823
  08:17:54 STEP 2607 H=2607:00 +CPU= 40.656
  08:18:26 STEP 2608 H=2608:00 +CPU= 27.920
  08:18:54 STEP 2609 H=2609:00 +CPU= 26.921
  08:19:35 STEP 2610 H=2610:00 +CPU= 41.321
  08:20:04 STEP 2611 H=2611:00 +CPU= 26.350
  08:20:33 STEP 2612 H=2612:00 +CPU= 27.304
  08:21:16 STEP 2613 H=2613:00 +CPU= 40.584
  08:21:43 STEP 2614 H=2614:00 +CPU= 26.561
  08:22:13 STEP 2615 H=2615:00 +CPU= 27.539
  08:23:02 STEP 2616 H=2616:00 +CPU= 47.376
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_108.zip
Uploading the intermediate file: upload_file_108.zip
  08:23:32 STEP 2617 H=2617:00 +CPU= 27.795
  08:24:03 STEP 2618 H=2618:00 +CPU= 29.094
  08:24:47 STEP 2619 H=2619:00 +CPU= 40.760
  08:25:15 STEP 2620 H=2620:00 +CPU= 27.096
  08:25:46 STEP 2621 H=2621:00 +CPU= 27.004
  08:26:30 STEP 2622 H=2622:00 +CPU= 40.578
  08:26:58 STEP 2623 H=2623:00 +CPU= 26.761
  08:27:31 STEP 2624 H=2624:00 +CPU= 28.597
  08:28:14 STEP 2625 H=2625:00 +CPU= 40.651
  08:28:46 STEP 2626 H=2626:00 +CPU= 28.152
  08:29:18 STEP 2627 H=2627:00 +CPU= 27.868
  08:30:10 STEP 2628 H=2628:00 +CPU= 48.688
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002628
  08:30:42 STEP 2629 H=2629:00 +CPU= 29.335
  08:31:15 STEP 2630 H=2630:00 +CPU= 29.212
  08:31:59 STEP 2631 H=2631:00 +CPU= 41.115
  08:32:24 STEP 2632 H=2632:00 +CPU= 25.537
  08:32:55 STEP 2633 H=2633:00 +CPU= 27.264
  08:33:39 STEP 2634 H=2634:00 +CPU= 41.993
  08:34:08 STEP 2635 H=2635:00 +CPU= 26.745
  08:34:38 STEP 2636 H=2636:00 +CPU= 27.815
  08:35:22 STEP 2637 H=2637:00 +CPU= 40.841
  08:35:52 STEP 2638 H=2638:00 +CPU= 27.849
  08:36:23 STEP 2639 H=2639:00 +CPU= 29.293
  08:37:14 STEP 2640 H=2640:00 +CPU= 49.140
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_109.zip
Uploading the intermediate file: upload_file_109.zip
  08:37:43 STEP 2641 H=2641:00 +CPU= 27.120
  08:38:12 STEP 2642 H=2642:00 +CPU= 27.341
  08:38:58 STEP 2643 H=2643:00 +CPU= 42.382
  08:39:27 STEP 2644 H=2644:00 +CPU= 27.595
  08:39:56 STEP 2645 H=2645:00 +CPU= 27.863
  08:40:41 STEP 2646 H=2646:00 +CPU= 42.597
  08:41:11 STEP 2647 H=2647:00 +CPU= 26.995
  08:41:41 STEP 2648 H=2648:00 +CPU= 28.152
  08:42:25 STEP 2649 H=2649:00 +CPU= 41.656
  08:42:55 STEP 2650 H=2650:00 +CPU= 28.174
  08:43:26 STEP 2651 H=2651:00 +CPU= 27.842
  08:44:13 STEP 2652 H=2652:00 +CPU= 44.802
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002652
  08:44:44 STEP 2653 H=2653:00 +CPU= 27.694
  08:45:15 STEP 2654 H=2654:00 +CPU= 27.557
  08:45:57 STEP 2655 H=2655:00 +CPU= 39.725
  08:46:27 STEP 2656 H=2656:00 +CPU= 28.135
  08:47:00 STEP 2657 H=2657:00 +CPU= 28.795
  08:47:44 STEP 2658 H=2658:00 +CPU= 43.458
  08:48:14 STEP 2659 H=2659:00 +CPU= 27.629
  08:48:40 STEP 2660 H=2660:00 +CPU= 26.085
  08:49:23 STEP 2661 H=2661:00 +CPU= 41.909
  08:49:54 STEP 2662 H=2662:00 +CPU= 28.846
  08:50:23 STEP 2663 H=2663:00 +CPU= 28.448
  08:51:17 STEP 2664 H=2664:00 +CPU= 52.310
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194639/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0995_2008050100_123_977_12194639_0_r1797057869_110.zip
Uploading the intermediate file: upload_file_110.zip
  08:51:46 STEP 2665 H=2665:00 +CPU= 27.562
  08:52:15 STEP 2666 H=2666:00 +CPU= 27.770
  08:53:03 STEP 2667 H=2667:00 +CPU= 42.919
  08:53:35 STEP 2668 H=2668:00 +CPU= 29.622
  08:54:04 STEP 2669 H=2669:00 +CPU= 28.029
  08:54:55 STEP 2670 H=2670:00 +CPU= 49.054
  08:55:32 STEP 2671 H=2671:00 +CPU= 36.680
  08:56:04 STEP 2672 H=2672:00 +CPU= 29.375
  08:56:48 STEP 2673 H=2673:00 +CPU= 42.392
  08:57:20 STEP 2674 H=2674:00 +CPU= 29.313
  08:57:51 STEP 2675 H=2675:00 +CPU= 29.371
  08:58:43 STEP 2676 H=2676:00 +CPU= 47.868
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+002676
  08:59:15 STEP 2677 H=2677:00 +CPU= 28.250
  08:59:45 STEP 2678 H=2678:00 +CPU= 28.810
  09:00:29 STEP 2679 H=2679:00 +CPU= 41.568
  09:00:59 STEP 2680 H=2680:00 +CPU= 28.276
  09:01:30 STEP 2681 H=2681:00 +CPU= 28.536
  09:02:16 STEP 2682 H=2682:00 +CPU= 43.104
  09:02:46 STEP 2683 H=2683:00 +CPU= 27.876
  09:03:18 STEP 2684 H=2684:00 +CPU= 29.040
  09:04:05 STEP 2685 H=2685:00 +CPU= 43.060
  09:04:36 STEP 2686 H=2686:00 +CPU= 29.654
  09:05:08 STEP 2687 H=2687:00 +CPU= 29.492
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:23 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 9,072,000 62,763 0.0069
09 May 2023 16:05:18 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 7,603,200 50,725 0.0067
09 May 2023 15:53:38 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 5,443,200 37,793 0.0069
09 May 2023 15:53:08 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 4,579,200 32,018 0.0070
09 May 2023 15:52:55 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 3,715,200 26,308 0.0071
09 May 2023 15:52:43 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 2,851,200 20,035 0.0070
09 May 2023 15:52:30 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 1,987,200 13,414 0.0068
09 May 2023 15:52:02 1520317 22280498 oifs_43r3_ps_0995_2008050100_123_977_12194639_0 864,000 5,900 0.0068


©2024 climateprediction.net