climateprediction.net home page
Task 22281346

Task 22281346

Name oifs_43r3_ps_0782_2009050100_123_978_12195426_0
Workunit 12195426
Created 23 Dec 2022, 21:57:08 UTC
Sent 23 Dec 2022, 22:10:37 UTC
Report deadline 22 Jan 2023, 22:10:37 UTC
Received 11 Jan 2023, 20:07:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537350
Run time 21 hours 32 min 12 sec
CPU time 21 hours 28 min 34 sec
Validate state Invalid
Credit 13,305.60
Device peak FLOPS 5.57 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.21 MB
Peak swap size 4,978.07 MB
Peak disk usage 1,488.56 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
mateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_90.zip
Uploading the intermediate file: upload_file_90.zip
  15:34:01 STEP 2185 H=2185:00 +CPU= 25.873
  15:34:27 STEP 2186 H=2186:00 +CPU= 25.911
  15:35:03 STEP 2187 H=2187:00 +CPU= 35.865
  15:35:28 STEP 2188 H=2188:00 +CPU= 25.030
  15:35:49 STEP 2189 H=2189:00 +CPU= 20.459
  15:36:27 STEP 2190 H=2190:00 +CPU= 38.488
  15:36:52 STEP 2191 H=2191:00 +CPU= 25.178
  15:37:19 STEP 2192 H=2192:00 +CPU= 27.007
  15:37:59 STEP 2193 H=2193:00 +CPU= 39.598
  15:38:25 STEP 2194 H=2194:00 +CPU= 26.107
  15:38:50 STEP 2195 H=2195:00 +CPU= 24.904
  15:39:32 STEP 2196 H=2196:00 +CPU= 42.183
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002196
  15:39:58 STEP 2197 H=2197:00 +CPU= 25.880
  15:40:25 STEP 2198 H=2198:00 +CPU= 26.723
  15:41:03 STEP 2199 H=2199:00 +CPU= 37.674
  15:41:27 STEP 2200 H=2200:00 +CPU= 24.649
  15:41:52 STEP 2201 H=2201:00 +CPU= 24.531
  15:42:33 STEP 2202 H=2202:00 +CPU= 40.869
  15:42:57 STEP 2203 H=2203:00 +CPU= 24.209
  15:43:24 STEP 2204 H=2204:00 +CPU= 26.623
  15:44:03 STEP 2205 H=2205:00 +CPU= 39.196
  15:44:29 STEP 2206 H=2206:00 +CPU= 26.676
  15:44:56 STEP 2207 H=2207:00 +CPU= 26.347
  15:45:42 STEP 2208 H=2208:00 +CPU= 46.482
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_91.zip
Uploading the intermediate file: upload_file_91.zip
  15:46:04 STEP 2209 H=2209:00 +CPU= 21.389
  15:46:28 STEP 2210 H=2210:00 +CPU= 24.736
  15:47:03 STEP 2211 H=2211:00 +CPU= 34.501
  15:47:29 STEP 2212 H=2212:00 +CPU= 25.692
  15:47:53 STEP 2213 H=2213:00 +CPU= 24.196
  15:48:34 STEP 2214 H=2214:00 +CPU= 40.577
  15:48:58 STEP 2215 H=2215:00 +CPU= 24.129
  15:49:24 STEP 2216 H=2216:00 +CPU= 25.888
  15:50:02 STEP 2217 H=2217:00 +CPU= 38.809
  15:50:29 STEP 2218 H=2218:00 +CPU= 26.630
  15:50:54 STEP 2219 H=2219:00 +CPU= 24.892
  15:51:38 STEP 2220 H=2220:00 +CPU= 44.148
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002220
  15:52:03 STEP 2221 H=2221:00 +CPU= 24.148
  15:52:28 STEP 2222 H=2222:00 +CPU= 25.019
  15:53:07 STEP 2223 H=2223:00 +CPU= 39.622
  15:53:33 STEP 2224 H=2224:00 +CPU= 25.838
  15:53:59 STEP 2225 H=2225:00 +CPU= 25.625
  15:54:39 STEP 2226 H=2226:00 +CPU= 39.921
  15:55:03 STEP 2227 H=2227:00 +CPU= 24.716
  15:55:30 STEP 2228 H=2228:00 +CPU= 26.258
  15:56:09 STEP 2229 H=2229:00 +CPU= 39.065
  15:56:35 STEP 2230 H=2230:00 +CPU= 26.207
  15:56:59 STEP 2231 H=2231:00 +CPU= 24.264
  15:57:45 STEP 2232 H=2232:00 +CPU= 45.728
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_92.zip
Uploading the intermediate file: upload_file_92.zip
  15:58:10 STEP 2233 H=2233:00 +CPU= 24.720
  15:58:36 STEP 2234 H=2234:00 +CPU= 26.098
  15:59:12 STEP 2235 H=2235:00 +CPU= 35.668
  15:59:38 STEP 2236 H=2236:00 +CPU= 26.277
  16:00:03 STEP 2237 H=2237:00 +CPU= 25.334
  16:00:44 STEP 2238 H=2238:00 +CPU= 40.336
  16:01:09 STEP 2239 H=2239:00 +CPU= 25.570
  16:01:36 STEP 2240 H=2240:00 +CPU= 26.457
  16:02:15 STEP 2241 H=2241:00 +CPU= 38.917
  16:02:40 STEP 2242 H=2242:00 +CPU= 25.841
  16:03:01 STEP 2243 H=2243:00 +CPU= 20.165
  16:03:45 STEP 2244 H=2244:00 +CPU= 44.507
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002244
  16:04:10 STEP 2245 H=2245:00 +CPU= 24.612
  16:04:32 STEP 2246 H=2246:00 +CPU= 22.535
  16:05:11 STEP 2247 H=2247:00 +CPU= 39.002
  16:05:37 STEP 2248 H=2248:00 +CPU= 25.455
  16:06:01 STEP 2249 H=2249:00 +CPU= 24.207
  16:06:40 STEP 2250 H=2250:00 +CPU= 38.579
  16:07:05 STEP 2251 H=2251:00 +CPU= 24.942
  16:07:29 STEP 2252 H=2252:00 +CPU= 24.362
  16:08:08 STEP 2253 H=2253:00 +CPU= 38.857
  16:08:34 STEP 2254 H=2254:00 +CPU= 26.002
  16:08:57 STEP 2255 H=2255:00 +CPU= 22.996
  16:09:43 STEP 2256 H=2256:00 +CPU= 45.873
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_93.zip
Uploading the intermediate file: upload_file_93.zip
  16:10:07 STEP 2257 H=2257:00 +CPU= 24.012
  16:10:34 STEP 2258 H=2258:00 +CPU= 26.755
  16:11:11 STEP 2259 H=2259:00 +CPU= 37.522
  16:11:38 STEP 2260 H=2260:00 +CPU= 26.222
  16:12:03 STEP 2261 H=2261:00 +CPU= 25.066
  16:12:38 STEP 2262 H=2262:00 +CPU= 35.881
  16:13:02 STEP 2263 H=2263:00 +CPU= 23.171
  16:13:28 STEP 2264 H=2264:00 +CPU= 26.533
  16:14:08 STEP 2265 H=2265:00 +CPU= 39.331
  16:14:33 STEP 2266 H=2266:00 +CPU= 25.909
  16:14:57 STEP 2267 H=2267:00 +CPU= 23.222
  16:15:40 STEP 2268 H=2268:00 +CPU= 42.835
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002268
  16:16:05 STEP 2269 H=2269:00 +CPU= 25.813
  16:16:31 STEP 2270 H=2270:00 +CPU= 25.837
  16:17:10 STEP 2271 H=2271:00 +CPU= 38.624
  16:17:35 STEP 2272 H=2272:00 +CPU= 25.284
  16:18:00 STEP 2273 H=2273:00 +CPU= 24.978
  16:18:36 STEP 2274 H=2274:00 +CPU= 35.469
  16:19:01 STEP 2275 H=2275:00 +CPU= 25.256
  16:19:27 STEP 2276 H=2276:00 +CPU= 25.988
  16:20:03 STEP 2277 H=2277:00 +CPU= 36.229
  16:20:30 STEP 2278 H=2278:00 +CPU= 26.671
  16:20:55 STEP 2279 H=2279:00 +CPU= 25.445
  16:21:40 STEP 2280 H=2280:00 +CPU= 44.303
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_94.zip
Uploading the intermediate file: upload_file_94.zip
  16:22:02 STEP 2281 H=2281:00 +CPU= 22.439
  16:22:28 STEP 2282 H=2282:00 +CPU= 26.219
  16:23:04 STEP 2283 H=2283:00 +CPU= 35.839
  16:23:30 STEP 2284 H=2284:00 +CPU= 25.965
  16:23:54 STEP 2285 H=2285:00 +CPU= 23.688
  16:24:34 STEP 2286 H=2286:00 +CPU= 39.739
  16:24:57 STEP 2287 H=2287:00 +CPU= 23.545
  16:25:23 STEP 2288 H=2288:00 +CPU= 25.480
  16:26:02 STEP 2289 H=2289:00 +CPU= 38.784
  16:26:27 STEP 2290 H=2290:00 +CPU= 25.759
  16:26:52 STEP 2291 H=2291:00 +CPU= 24.741
  16:27:34 STEP 2292 H=2292:00 +CPU= 41.731
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002292
  16:28:00 STEP 2293 H=2293:00 +CPU= 25.736
  16:28:25 STEP 2294 H=2294:00 +CPU= 25.781
  16:29:04 STEP 2295 H=2295:00 +CPU= 38.287
  16:29:30 STEP 2296 H=2296:00 +CPU= 25.957
  16:29:55 STEP 2297 H=2297:00 +CPU= 25.323
  16:30:34 STEP 2298 H=2298:00 +CPU= 38.994
  16:30:58 STEP 2299 H=2299:00 +CPU= 23.773
  16:31:24 STEP 2300 H=2300:00 +CPU= 25.649
  16:32:03 STEP 2301 H=2301:00 +CPU= 38.993
  16:32:29 STEP 2302 H=2302:00 +CPU= 26.774
  16:32:55 STEP 2303 H=2303:00 +CPU= 25.330
  16:33:41 STEP 2304 H=2304:00 +CPU= 46.190
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_95.zip
Uploading the intermediate file: upload_file_95.zip
  16:34:06 STEP 2305 H=2305:00 +CPU= 25.289
  16:34:33 STEP 2306 H=2306:00 +CPU= 26.546
  16:35:12 STEP 2307 H=2307:00 +CPU= 38.779
  16:35:38 STEP 2308 H=2308:00 +CPU= 26.620
  16:36:00 STEP 2309 H=2309:00 +CPU= 22.153
  16:36:40 STEP 2310 H=2310:00 +CPU= 39.587
  16:37:03 STEP 2311 H=2311:00 +CPU= 22.509
  16:37:26 STEP 2312 H=2312:00 +CPU= 23.412
  16:38:04 STEP 2313 H=2313:00 +CPU= 38.144
  16:38:31 STEP 2314 H=2314:00 +CPU= 26.931
  16:38:52 STEP 2315 H=2315:00 +CPU= 21.327
  16:39:35 STEP 2316 H=2316:00 +CPU= 42.639
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002316
  16:40:00 STEP 2317 H=2317:00 +CPU= 25.080
  16:40:26 STEP 2318 H=2318:00 +CPU= 25.891
  16:41:06 STEP 2319 H=2319:00 +CPU= 39.336
  16:41:32 STEP 2320 H=2320:00 +CPU= 26.596
  16:41:56 STEP 2321 H=2321:00 +CPU= 24.078
  16:42:37 STEP 2322 H=2322:00 +CPU= 40.441
  16:43:01 STEP 2323 H=2323:00 +CPU= 24.789
  16:43:27 STEP 2324 H=2324:00 +CPU= 25.798
  16:44:04 STEP 2325 H=2325:00 +CPU= 37.139
  16:44:29 STEP 2326 H=2326:00 +CPU= 25.014
  16:44:55 STEP 2327 H=2327:00 +CPU= 25.785
  16:45:41 STEP 2328 H=2328:00 +CPU= 45.298
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_96.zip
Uploading the intermediate file: upload_file_96.zip
  16:46:05 STEP 2329 H=2329:00 +CPU= 24.958
  16:46:28 STEP 2330 H=2330:00 +CPU= 22.801
  16:47:07 STEP 2331 H=2331:00 +CPU= 38.625
  16:47:31 STEP 2332 H=2332:00 +CPU= 23.946
  16:47:56 STEP 2333 H=2333:00 +CPU= 24.620
  16:48:35 STEP 2334 H=2334:00 +CPU= 39.676
  16:49:00 STEP 2335 H=2335:00 +CPU= 24.648
  16:49:23 STEP 2336 H=2336:00 +CPU= 23.409
  16:50:03 STEP 2337 H=2337:00 +CPU= 39.445
  16:50:29 STEP 2338 H=2338:00 +CPU= 26.378
  16:50:54 STEP 2339 H=2339:00 +CPU= 24.693
  16:51:38 STEP 2340 H=2340:00 +CPU= 44.553
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002340
  16:52:04 STEP 2341 H=2341:00 +CPU= 25.230
  16:52:30 STEP 2342 H=2342:00 +CPU= 25.783
  16:53:10 STEP 2343 H=2343:00 +CPU= 40.361
  16:53:35 STEP 2344 H=2344:00 +CPU= 25.117
  16:53:59 STEP 2345 H=2345:00 +CPU= 24.251
  16:54:40 STEP 2346 H=2346:00 +CPU= 40.138
  16:55:05 STEP 2347 H=2347:00 +CPU= 25.652
  16:55:31 STEP 2348 H=2348:00 +CPU= 26.180
  16:56:11 STEP 2349 H=2349:00 +CPU= 39.886
  16:56:38 STEP 2350 H=2350:00 +CPU= 26.267
  16:57:01 STEP 2351 H=2351:00 +CPU= 23.679
  16:57:45 STEP 2352 H=2352:00 +CPU= 43.135
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_97.zip
Uploading the intermediate file: upload_file_97.zip
  16:58:08 STEP 2353 H=2353:00 +CPU= 23.426
  16:58:34 STEP 2354 H=2354:00 +CPU= 25.984
  16:59:11 STEP 2355 H=2355:00 +CPU= 37.333
  16:59:36 STEP 2356 H=2356:00 +CPU= 24.536
  17:00:01 STEP 2357 H=2357:00 +CPU= 25.209
  17:00:42 STEP 2358 H=2358:00 +CPU= 40.485
  17:01:07 STEP 2359 H=2359:00 +CPU= 25.525
  17:01:34 STEP 2360 H=2360:00 +CPU= 26.271
  17:02:13 STEP 2361 H=2361:00 +CPU= 39.382
  17:02:38 STEP 2362 H=2362:00 +CPU= 24.822
  17:03:02 STEP 2363 H=2363:00 +CPU= 24.121
  17:03:45 STEP 2364 H=2364:00 +CPU= 43.161
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002364
  17:04:10 STEP 2365 H=2365:00 +CPU= 24.842
  17:04:36 STEP 2366 H=2366:00 +CPU= 26.022
  17:05:12 STEP 2367 H=2367:00 +CPU= 35.584
  17:05:38 STEP 2368 H=2368:00 +CPU= 25.705
  17:06:00 STEP 2369 H=2369:00 +CPU= 22.569
  17:06:39 STEP 2370 H=2370:00 +CPU= 38.380
  17:07:02 STEP 2371 H=2371:00 +CPU= 23.489
  17:07:29 STEP 2372 H=2372:00 +CPU= 26.704
  17:08:09 STEP 2373 H=2373:00 +CPU= 40.317
  17:08:35 STEP 2374 H=2374:00 +CPU= 26.254
  17:09:01 STEP 2375 H=2375:00 +CPU= 25.956
  17:09:48 STEP 2376 H=2376:00 +CPU= 46.887
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_98.zip
Uploading the intermediate file: upload_file_98.zip
  17:10:13 STEP 2377 H=2377:00 +CPU= 25.241
  17:10:36 STEP 2378 H=2378:00 +CPU= 22.257
  17:11:14 STEP 2379 H=2379:00 +CPU= 38.014
  17:11:39 STEP 2380 H=2380:00 +CPU= 25.563
  17:12:05 STEP 2381 H=2381:00 +CPU= 25.743
  17:12:46 STEP 2382 H=2382:00 +CPU= 40.658
  17:13:08 STEP 2383 H=2383:00 +CPU= 21.832
  17:13:32 STEP 2384 H=2384:00 +CPU= 24.263
  17:14:10 STEP 2385 H=2385:00 +CPU= 38.458
  17:14:36 STEP 2386 H=2386:00 +CPU= 25.973
  17:14:59 STEP 2387 H=2387:00 +CPU= 22.195
  17:15:41 STEP 2388 H=2388:00 +CPU= 42.512
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002388
  17:16:06 STEP 2389 H=2389:00 +CPU= 24.409
  17:16:32 STEP 2390 H=2390:00 +CPU= 26.293
  17:17:11 STEP 2391 H=2391:00 +CPU= 39.506
  17:17:37 STEP 2392 H=2392:00 +CPU= 25.489
  17:18:02 STEP 2393 H=2393:00 +CPU= 24.877
  17:18:42 STEP 2394 H=2394:00 +CPU= 40.163
  17:19:06 STEP 2395 H=2395:00 +CPU= 24.388
  17:19:32 STEP 2396 H=2396:00 +CPU= 25.222
  17:20:10 STEP 2397 H=2397:00 +CPU= 38.346
  17:20:36 STEP 2398 H=2398:00 +CPU= 25.986
  17:21:01 STEP 2399 H=2399:00 +CPU= 25.317
  17:21:45 STEP 2400 H=2400:00 +CPU= 43.535
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_99.zip
Uploading the intermediate file: upload_file_99.zip
Uploading trickle at timestep: 8640000
  17:22:10 STEP 2401 H=2401:00 +CPU= 25.013
  17:22:37 STEP 2402 H=2402:00 +CPU= 26.828
  17:23:17 STEP 2403 H=2403:00 +CPU= 39.607
  17:23:41 STEP 2404 H=2404:00 +CPU= 24.801
  17:24:05 STEP 2405 H=2405:00 +CPU= 23.841
  17:24:45 STEP 2406 H=2406:00 +CPU= 39.428
  17:25:09 STEP 2407 H=2407:00 +CPU= 24.126
  17:25:35 STEP 2408 H=2408:00 +CPU= 26.618
  17:26:16 STEP 2409 H=2409:00 +CPU= 40.338
  17:26:41 STEP 2410 H=2410:00 +CPU= 25.419
  17:27:05 STEP 2411 H=2411:00 +CPU= 23.469
  17:27:46 STEP 2412 H=2412:00 +CPU= 41.484
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002412
  17:28:10 STEP 2413 H=2413:00 +CPU= 24.176
  17:28:35 STEP 2414 H=2414:00 +CPU= 24.864
  17:29:10 STEP 2415 H=2415:00 +CPU= 34.634
  17:29:35 STEP 2416 H=2416:00 +CPU= 25.407
  17:30:01 STEP 2417 H=2417:00 +CPU= 25.389
  17:30:41 STEP 2418 H=2418:00 +CPU= 40.265
  17:31:05 STEP 2419 H=2419:00 +CPU= 24.146
  17:31:31 STEP 2420 H=2420:00 +CPU= 25.991
  17:32:10 STEP 2421 H=2421:00 +CPU= 39.054
  17:32:32 STEP 2422 H=2422:00 +CPU= 21.750
  17:32:56 STEP 2423 H=2423:00 +CPU= 23.582
  17:33:41 STEP 2424 H=2424:00 +CPU= 45.492
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_100.zip
Uploading the intermediate file: upload_file_100.zip
  17:34:06 STEP 2425 H=2425:00 +CPU= 25.100
  17:34:33 STEP 2426 H=2426:00 +CPU= 26.438
  17:35:12 STEP 2427 H=2427:00 +CPU= 39.229
  17:35:37 STEP 2428 H=2428:00 +CPU= 25.544
  17:36:03 STEP 2429 H=2429:00 +CPU= 25.473
  17:36:42 STEP 2430 H=2430:00 +CPU= 39.530
  17:37:07 STEP 2431 H=2431:00 +CPU= 24.618
  17:37:34 STEP 2432 H=2432:00 +CPU= 26.766
  17:38:11 STEP 2433 H=2433:00 +CPU= 36.761
  17:38:35 STEP 2434 H=2434:00 +CPU= 23.847
  17:38:59 STEP 2435 H=2435:00 +CPU= 24.386
  17:39:39 STEP 2436 H=2436:00 +CPU= 40.074
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002436
  17:40:04 STEP 2437 H=2437:00 +CPU= 25.048
  17:40:28 STEP 2438 H=2438:00 +CPU= 23.569
  17:41:06 STEP 2439 H=2439:00 +CPU= 38.206
  17:41:30 STEP 2440 H=2440:00 +CPU= 23.744
  17:41:54 STEP 2441 H=2441:00 +CPU= 23.665
  17:42:30 STEP 2442 H=2442:00 +CPU= 36.059
  17:42:55 STEP 2443 H=2443:00 +CPU= 25.631
  17:43:21 STEP 2444 H=2444:00 +CPU= 25.885
  17:44:00 STEP 2445 H=2445:00 +CPU= 39.060
  17:44:26 STEP 2446 H=2446:00 +CPU= 25.909
  17:44:51 STEP 2447 H=2447:00 +CPU= 24.767
  17:45:38 STEP 2448 H=2448:00 +CPU= 46.720
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_101.zip
Uploading the intermediate file: upload_file_101.zip
  17:46:03 STEP 2449 H=2449:00 +CPU= 24.911
  17:46:26 STEP 2450 H=2450:00 +CPU= 22.958
  17:47:02 STEP 2451 H=2451:00 +CPU= 36.336
  17:47:28 STEP 2452 H=2452:00 +CPU= 26.214
  17:47:53 STEP 2453 H=2453:00 +CPU= 25.227
  17:48:33 STEP 2454 H=2454:00 +CPU= 39.682
  17:48:57 STEP 2455 H=2455:00 +CPU= 23.819
  17:49:22 STEP 2456 H=2456:00 +CPU= 25.238
  17:50:00 STEP 2457 H=2457:00 +CPU= 38.183
  17:50:27 STEP 2458 H=2458:00 +CPU= 26.193
  17:50:52 STEP 2459 H=2459:00 +CPU= 25.516
  17:51:34 STEP 2460 H=2460:00 +CPU= 41.535
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002460
  17:51:59 STEP 2461 H=2461:00 +CPU= 25.563
  17:52:25 STEP 2462 H=2462:00 +CPU= 26.133
  17:53:05 STEP 2463 H=2463:00 +CPU= 39.245
  17:53:30 STEP 2464 H=2464:00 +CPU= 25.835
  17:53:56 STEP 2465 H=2465:00 +CPU= 25.752
  17:54:36 STEP 2466 H=2466:00 +CPU= 40.279
  17:55:01 STEP 2467 H=2467:00 +CPU= 25.001
  17:55:28 STEP 2468 H=2468:00 +CPU= 26.355
  17:56:05 STEP 2469 H=2469:00 +CPU= 37.560
  17:56:30 STEP 2470 H=2470:00 +CPU= 24.309
  17:56:55 STEP 2471 H=2471:00 +CPU= 25.660
  17:57:42 STEP 2472 H=2472:00 +CPU= 46.176
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_102.zip
Uploading the intermediate file: upload_file_102.zip
  17:58:03 STEP 2473 H=2473:00 +CPU= 21.811
  17:58:24 STEP 2474 H=2474:00 +CPU= 20.176
  17:59:02 STEP 2475 H=2475:00 +CPU= 37.908
  17:59:28 STEP 2476 H=2476:00 +CPU= 26.732
  17:59:54 STEP 2477 H=2477:00 +CPU= 25.821
  18:00:34 STEP 2478 H=2478:00 +CPU= 40.093
  18:00:59 STEP 2479 H=2479:00 +CPU= 24.861
  18:01:26 STEP 2480 H=2480:00 +CPU= 26.882
  18:02:04 STEP 2481 H=2481:00 +CPU= 37.912
  18:02:30 STEP 2482 H=2482:00 +CPU= 26.337
  18:02:55 STEP 2483 H=2483:00 +CPU= 24.827
  18:03:39 STEP 2484 H=2484:00 +CPU= 43.234
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002484
  18:04:02 STEP 2485 H=2485:00 +CPU= 23.511
  18:04:29 STEP 2486 H=2486:00 +CPU= 26.501
  18:05:08 STEP 2487 H=2487:00 +CPU= 39.377
  18:05:33 STEP 2488 H=2488:00 +CPU= 24.898
  18:05:59 STEP 2489 H=2489:00 +CPU= 26.175
  18:06:40 STEP 2490 H=2490:00 +CPU= 40.990
  18:07:05 STEP 2491 H=2491:00 +CPU= 24.832
  18:07:32 STEP 2492 H=2492:00 +CPU= 27.227
  18:08:09 STEP 2493 H=2493:00 +CPU= 36.463
  18:08:34 STEP 2494 H=2494:00 +CPU= 25.301
  18:08:57 STEP 2495 H=2495:00 +CPU= 22.825
  18:09:39 STEP 2496 H=2496:00 +CPU= 42.041
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_103.zip
Uploading the intermediate file: upload_file_103.zip
  18:10:01 STEP 2497 H=2497:00 +CPU= 22.492
  18:10:27 STEP 2498 H=2498:00 +CPU= 25.365
  18:11:04 STEP 2499 H=2499:00 +CPU= 37.169
  18:11:30 STEP 2500 H=2500:00 +CPU= 25.937
  18:11:56 STEP 2501 H=2501:00 +CPU= 25.812
  18:12:36 STEP 2502 H=2502:00 +CPU= 40.222
  18:13:01 STEP 2503 H=2503:00 +CPU= 24.741
  18:13:27 STEP 2504 H=2504:00 +CPU= 26.498
  18:14:05 STEP 2505 H=2505:00 +CPU= 37.930
  18:14:30 STEP 2506 H=2506:00 +CPU= 24.876
  18:14:55 STEP 2507 H=2507:00 +CPU= 25.012
  18:15:39 STEP 2508 H=2508:00 +CPU= 43.620
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002508
  18:16:03 STEP 2509 H=2509:00 +CPU= 24.789
  18:16:29 STEP 2510 H=2510:00 +CPU= 26.090
  18:17:07 STEP 2511 H=2511:00 +CPU= 37.721
  18:17:29 STEP 2512 H=2512:00 +CPU= 21.637
  18:17:53 STEP 2513 H=2513:00 +CPU= 24.539
  18:18:32 STEP 2514 H=2514:00 +CPU= 38.436
  18:18:57 STEP 2515 H=2515:00 +CPU= 24.617
  18:19:23 STEP 2516 H=2516:00 +CPU= 26.576
  18:20:03 STEP 2517 H=2517:00 +CPU= 39.716
  18:20:29 STEP 2518 H=2518:00 +CPU= 25.763
  18:20:55 STEP 2519 H=2519:00 +CPU= 25.871
  18:21:39 STEP 2520 H=2520:00 +CPU= 44.245
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_104.zip
Uploading the intermediate file: upload_file_104.zip
  18:22:04 STEP 2521 H=2521:00 +CPU= 25.293
  18:22:30 STEP 2522 H=2522:00 +CPU= 26.077
  18:23:08 STEP 2523 H=2523:00 +CPU= 38.161
  18:23:33 STEP 2524 H=2524:00 +CPU= 24.246
  18:23:58 STEP 2525 H=2525:00 +CPU= 24.854
  18:24:38 STEP 2526 H=2526:00 +CPU= 40.340
  18:25:03 STEP 2527 H=2527:00 +CPU= 24.662
  18:25:28 STEP 2528 H=2528:00 +CPU= 25.380
  18:26:06 STEP 2529 H=2529:00 +CPU= 37.584
  18:26:30 STEP 2530 H=2530:00 +CPU= 24.724
  18:26:53 STEP 2531 H=2531:00 +CPU= 22.531
  18:27:33 STEP 2532 H=2532:00 +CPU= 40.189
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002532
  18:27:58 STEP 2533 H=2533:00 +CPU= 25.321
  18:28:24 STEP 2534 H=2534:00 +CPU= 25.617
  18:29:01 STEP 2535 H=2535:00 +CPU= 36.872
  18:29:27 STEP 2536 H=2536:00 +CPU= 25.936
  18:29:53 STEP 2537 H=2537:00 +CPU= 25.701
  18:30:33 STEP 2538 H=2538:00 +CPU= 40.796
  18:30:57 STEP 2539 H=2539:00 +CPU= 23.545
  18:31:23 STEP 2540 H=2540:00 +CPU= 25.886
  18:32:02 STEP 2541 H=2541:00 +CPU= 39.080
  18:32:27 STEP 2542 H=2542:00 +CPU= 24.694
  18:32:52 STEP 2543 H=2543:00 +CPU= 25.267
  18:33:38 STEP 2544 H=2544:00 +CPU= 46.057
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:34:00 STEP 2545 H=2545:00 +CPU= 22.322
  18:34:22 STEP 2546 H=2546:00 +CPU= 21.335
  18:34:55 STEP 2547 H=2547:00 +CPU= 33.817
  18:35:15 STEP 2548 H=2548:00 +CPU= 19.305
  18:35:37 STEP 2549 H=2549:00 +CPU= 22.699
  18:36:13 STEP 2550 H=2550:00 +CPU= 35.955
  18:36:36 STEP 2551 H=2551:00 +CPU= 22.249
  18:36:55 STEP 2552 H=2552:00 +CPU= 18.976
  18:37:29 STEP 2553 H=2553:00 +CPU= 34.439
  18:37:52 STEP 2554 H=2554:00 +CPU= 22.858
  18:38:16 STEP 2555 H=2555:00 +CPU= 23.361
  18:38:55 STEP 2556 H=2556:00 +CPU= 39.500
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002556
  18:39:19 STEP 2557 H=2557:00 +CPU= 24.363
  18:39:44 STEP 2558 H=2558:00 +CPU= 24.878
  18:40:19 STEP 2559 H=2559:00 +CPU= 34.947
  18:40:42 STEP 2560 H=2560:00 +CPU= 22.306
  18:41:05 STEP 2561 H=2561:00 +CPU= 23.282
  18:41:44 STEP 2562 H=2562:00 +CPU= 38.603
  18:42:06 STEP 2563 H=2563:00 +CPU= 22.299
  18:42:28 STEP 2564 H=2564:00 +CPU= 21.790
  18:43:06 STEP 2565 H=2565:00 +CPU= 38.219
  18:43:32 STEP 2566 H=2566:00 +CPU= 25.519
  18:43:53 STEP 2567 H=2567:00 +CPU= 21.276
  18:44:40 STEP 2568 H=2568:00 +CPU= 46.842
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:45:05 STEP 2569 H=2569:00 +CPU= 25.646
  18:45:31 STEP 2570 H=2570:00 +CPU= 25.488
  18:46:11 STEP 2571 H=2571:00 +CPU= 40.181
  18:46:38 STEP 2572 H=2572:00 +CPU= 26.495
  18:47:02 STEP 2573 H=2573:00 +CPU= 24.564
  18:47:43 STEP 2574 H=2574:00 +CPU= 40.839
  18:48:08 STEP 2575 H=2575:00 +CPU= 25.506
  18:48:34 STEP 2576 H=2576:00 +CPU= 25.941
  18:49:14 STEP 2577 H=2577:00 +CPU= 39.397
  18:49:40 STEP 2578 H=2578:00 +CPU= 25.759
  18:50:04 STEP 2579 H=2579:00 +CPU= 24.758
  18:50:48 STEP 2580 H=2580:00 +CPU= 43.977
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002580
  18:51:12 STEP 2581 H=2581:00 +CPU= 24.070
  18:51:35 STEP 2582 H=2582:00 +CPU= 22.134
  18:52:13 STEP 2583 H=2583:00 +CPU= 37.817
  18:52:39 STEP 2584 H=2584:00 +CPU= 26.084
  18:53:01 STEP 2585 H=2585:00 +CPU= 22.855
  18:53:40 STEP 2586 H=2586:00 +CPU= 38.896
  18:54:06 STEP 2587 H=2587:00 +CPU= 25.227
  18:54:30 STEP 2588 H=2588:00 +CPU= 24.318
  18:55:08 STEP 2589 H=2589:00 +CPU= 37.575
  18:55:33 STEP 2590 H=2590:00 +CPU= 25.731
  18:55:59 STEP 2591 H=2591:00 +CPU= 26.138
  18:56:43 STEP 2592 H=2592:00 +CPU= 43.552
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:57:08 STEP 2593 H=2593:00 +CPU= 25.171
  18:57:34 STEP 2594 H=2594:00 +CPU= 25.573
  18:58:14 STEP 2595 H=2595:00 +CPU= 40.291
  18:58:40 STEP 2596 H=2596:00 +CPU= 25.716
  18:59:02 STEP 2597 H=2597:00 +CPU= 21.559
  18:59:38 STEP 2598 H=2598:00 +CPU= 36.684
  19:00:02 STEP 2599 H=2599:00 +CPU= 23.668
  19:00:27 STEP 2600 H=2600:00 +CPU= 25.537
  19:01:07 STEP 2601 H=2601:00 +CPU= 39.874
  19:01:33 STEP 2602 H=2602:00 +CPU= 26.115
  19:01:59 STEP 2603 H=2603:00 +CPU= 25.101
  19:02:43 STEP 2604 H=2604:00 +CPU= 44.700
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002604
  19:03:06 STEP 2605 H=2605:00 +CPU= 22.419
  19:03:32 STEP 2606 H=2606:00 +CPU= 25.696
  19:04:10 STEP 2607 H=2607:00 +CPU= 38.853
  19:04:37 STEP 2608 H=2608:00 +CPU= 26.489
  19:05:02 STEP 2609 H=2609:00 +CPU= 25.434
  19:05:39 STEP 2610 H=2610:00 +CPU= 36.811
  19:06:03 STEP 2611 H=2611:00 +CPU= 23.965
  19:06:28 STEP 2612 H=2612:00 +CPU= 25.241
  19:07:06 STEP 2613 H=2613:00 +CPU= 37.450
  19:07:31 STEP 2614 H=2614:00 +CPU= 25.029
  19:07:56 STEP 2615 H=2615:00 +CPU= 25.114
  19:08:40 STEP 2616 H=2616:00 +CPU= 43.817
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_108.zip
Uploading the intermediate file: upload_file_108.zip
  19:09:05 STEP 2617 H=2617:00 +CPU= 24.459
  19:09:29 STEP 2618 H=2618:00 +CPU= 24.016
  19:10:04 STEP 2619 H=2619:00 +CPU= 35.517
  19:10:27 STEP 2620 H=2620:00 +CPU= 22.882
  19:10:50 STEP 2621 H=2621:00 +CPU= 23.328
  19:11:30 STEP 2622 H=2622:00 +CPU= 39.387
  19:11:54 STEP 2623 H=2623:00 +CPU= 24.243
  19:12:19 STEP 2624 H=2624:00 +CPU= 25.330
  19:12:58 STEP 2625 H=2625:00 +CPU= 38.210
  19:13:23 STEP 2626 H=2626:00 +CPU= 25.040
  19:13:46 STEP 2627 H=2627:00 +CPU= 23.049
  19:14:28 STEP 2628 H=2628:00 +CPU= 42.399
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002628
  19:14:54 STEP 2629 H=2629:00 +CPU= 25.413
  19:15:20 STEP 2630 H=2630:00 +CPU= 26.423
  19:15:59 STEP 2631 H=2631:00 +CPU= 38.987
  19:16:26 STEP 2632 H=2632:00 +CPU= 26.994
  19:16:51 STEP 2633 H=2633:00 +CPU= 25.120
  19:17:31 STEP 2634 H=2634:00 +CPU= 40.244
  19:17:57 STEP 2635 H=2635:00 +CPU= 25.309
  19:18:23 STEP 2636 H=2636:00 +CPU= 25.827
  19:19:02 STEP 2637 H=2637:00 +CPU= 39.184
  19:19:28 STEP 2638 H=2638:00 +CPU= 26.496
  19:19:54 STEP 2639 H=2639:00 +CPU= 25.699
  19:20:39 STEP 2640 H=2640:00 +CPU= 45.004
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275_109.zip
Uploading the intermediate file: upload_file_109.zip
  19:21:02 STEP 2641 H=2641:00 +CPU= 23.192
Uploading trickle at timestep: 9504000
  19:21:27 STEP 2642 H=2642:00 +CPU= 25.159
  19:22:07 STEP 2643 H=2643:00 +CPU= 39.236
  19:22:32 STEP 2644 H=2644:00 +CPU= 25.287
  19:22:57 STEP 2645 H=2645:00 +CPU= 24.556
  19:23:35 STEP 2646 H=2646:00 +CPU= 38.440
  19:24:00 STEP 2647 H=2647:00 +CPU= 24.819
  19:24:25 STEP 2648 H=2648:00 +CPU= 25.673
  19:25:06 STEP 2649 H=2649:00 +CPU= 40.395
  19:25:33 STEP 2650 H=2650:00 +CPU= 27.364
  19:25:59 STEP 2651 H=2651:00 +CPU= 25.526
  19:26:42 STEP 2652 H=2652:00 +CPU= 42.670
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002652
  19:27:07 STEP 2653 H=2653:00 +CPU= 25.515
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.825] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4415MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.825] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.825] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.825] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.825] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4415 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               GPGEO 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                VERINT 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192716:1671910036:76584.826] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VERINT_DGEMM_2 
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192717:1671910037:76585.826] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:c1:1:1:4272:4272] [20221224:192717:1671910037:76585.826] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000828F4C  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081DEDF  dr_hook_util_             152  dr_hook_util.F90
oifs_43r3_model.e  0000000000402DB1  yomhook_mp_dr_hoo          59  yomhook.F90
oifs_43r3_model.e  0000000000A3EC39  verint_                   193  verint.F90
oifs_43r3_model.e  0000000000A3D57B  gpgeo_                    139  gpgeo.F90
oifs_43r3_model.e  0000000000E43C51  ec_phys_                  458  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0782
(argv4) batchid: 978
(argv5) wuid: 12195426
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195426
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_febef6a6fe300fa8aad2c9cc06113628 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0782_2009050100_123_978_12195426.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0782_2009050100_123_978_12195426.zip
ic_ancil_file: ic_ancil_12195426
ifsdata_file: ifsdata_12195426
climate_data_file: clim_data_12195426
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3e1ebfce6b28c80c452f6c712619fce1 to: /var/lib/boinc-client/slots/8/ic_ancil_12195426.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12195426.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2301e2dab29c70866183a6a2fc580570 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12195426.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12195426.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12195426.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12195426.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12195426.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12195426.xml
-- Model is restarting --
Adjusting last_iter, 2654, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12195426.xml
last_cpu_time: 76534
upload_file_number: 110
last_iter: 2639
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0782_2009050100_123_978_12195426_0_r1827723275
The child process has been launched with process id: 1160
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:c1:1:1:1160:1160] [20230111:185300:1673463180:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:54:46 STEP 2640 H=2640:00 +CPU= 48.701
  18:55:11 STEP 2641 H=2641:00 +CPU= 24.979
  18:55:38 STEP 2642 H=2642:00 +CPU= 27.503
  18:56:20 STEP 2643 H=2643:00 +CPU= 42.035
  18:56:48 STEP 2644 H=2644:00 +CPU= 27.552
  18:57:15 STEP 2645 H=2645:00 +CPU= 27.267
  18:57:58 STEP 2646 H=2646:00 +CPU= 42.918
  18:58:25 STEP 2647 H=2647:00 +CPU= 26.991
  18:58:53 STEP 2648 H=2648:00 +CPU= 28.116
  18:59:33 STEP 2649 H=2649:00 +CPU= 39.316
  18:59:59 STEP 2650 H=2650:00 +CPU= 26.348
  19:00:27 STEP 2651 H=2651:00 +CPU= 27.523
  19:01:14 STEP 2652 H=2652:00 +CPU= 47.560
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+002652
  19:01:41 STEP 2653 H=2653:00 +CPU= 26.836
  19:02:10 STEP 2654 H=2654:00 +CPU= 28.787
  19:02:49 STEP 2655 H=2655:00 +CPU= 38.718
  19:03:14 STEP 2656 H=2656:00 +CPU= 25.417
  19:03:41 STEP 2657 H=2657:00 +CPU= 26.646
  19:04:21 STEP 2658 H=2658:00 +CPU= 40.252
  19:04:47 STEP 2659 H=2659:00 +CPU= 25.632
  19:05:15 STEP 2660 H=2660:00 +CPU= 28.338
  19:05:57 STEP 2661 H=2661:00 +CPU= 42.438
  19:06:26 STEP 2662 H=2662:00 +CPU= 28.373
  19:06:54 STEP 2663 H=2663:00 +CPU= 27.701
ECCODES ERROR   :  IO ERROR: No space left on dev..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:53:02 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 9,504,000 76,143 0.0080
09 May 2023 15:52:50 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 8,640,000 69,010 0.0080
09 May 2023 15:52:37 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 7,776,000 61,800 0.0079
09 May 2023 15:52:31 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 6,912,000 54,601 0.0079
09 May 2023 15:52:19 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 6,048,000 47,443 0.0078
09 May 2023 15:52:08 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 5,184,000 40,236 0.0078
09 May 2023 15:51:56 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 4,320,000 32,977 0.0076
09 May 2023 15:51:46 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 3,456,000 25,709 0.0074
09 May 2023 15:51:35 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 2,592,000 18,532 0.0071
09 May 2023 15:51:24 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 1,728,000 11,456 0.0066
09 May 2023 15:51:14 1537350 22281346 oifs_43r3_ps_0782_2009050100_123_978_12195426_0 864,000 5,205 0.0060


©2024 climateprediction.net