climateprediction.net home page
Task 22294856

Task 22294856

Name oifs_43r3_ps_0014_2012050100_123_981_12197658_1
Workunit 12197658
Created 31 Dec 2022, 22:44:42 UTC
Sent 1 Jan 2023, 4:04:45 UTC
Report deadline 31 Jan 2023, 4:04:45 UTC
Received 2 Jan 2023, 4:49:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1515225
Run time 22 hours 48 min 56 sec
CPU time 22 hours 43 min 22 sec
Validate state Invalid
Credit 13,668.48
Device peak FLOPS 4.56 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.86 MB
Peak swap size 4,978.05 MB
Peak disk usage 3,618.34 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:16322:16322] [20230101:151919:1672615159:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:20:11 STEP 2232 H=2232:00 +CPU= 48.214
  15:20:38 STEP 2233 H=2233:00 +CPU= 26.759
  15:21:06 STEP 2234 H=2234:00 +CPU= 27.954
  15:21:47 STEP 2235 H=2235:00 +CPU= 41.759
  15:22:15 STEP 2236 H=2236:00 +CPU= 28.125
  15:22:43 STEP 2237 H=2237:00 +CPU= 27.039
  15:23:25 STEP 2238 H=2238:00 +CPU= 42.359
  15:23:52 STEP 2239 H=2239:00 +CPU= 27.018
  15:24:20 STEP 2240 H=2240:00 +CPU= 27.848
  15:25:01 STEP 2241 H=2241:00 +CPU= 41.471
  15:25:29 STEP 2242 H=2242:00 +CPU= 27.741
  15:25:57 STEP 2243 H=2243:00 +CPU= 27.280
  15:26:43 STEP 2244 H=2244:00 +CPU= 46.140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002244
  15:27:10 STEP 2245 H=2245:00 +CPU= 27.041
  15:27:38 STEP 2246 H=2246:00 +CPU= 28.128
  15:28:20 STEP 2247 H=2247:00 +CPU= 41.762
  15:28:48 STEP 2248 H=2248:00 +CPU= 28.096
  15:29:15 STEP 2249 H=2249:00 +CPU= 26.886
  15:29:57 STEP 2250 H=2250:00 +CPU= 42.357
  15:30:24 STEP 2251 H=2251:00 +CPU= 26.735
  15:30:52 STEP 2252 H=2252:00 +CPU= 27.692
  15:31:33 STEP 2253 H=2253:00 +CPU= 41.713
  15:32:01 STEP 2254 H=2254:00 +CPU= 27.724
  15:32:29 STEP 2255 H=2255:00 +CPU= 28.057
  15:33:20 STEP 2256 H=2256:00 +CPU= 49.218
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_93.zip
Uploading the intermediate file: upload_file_93.zip
  15:33:47 STEP 2257 H=2257:00 +CPU= 27.888
  15:34:16 STEP 2258 H=2258:00 +CPU= 28.386
  15:34:58 STEP 2259 H=2259:00 +CPU= 41.968
  15:35:26 STEP 2260 H=2260:00 +CPU= 28.486
  15:35:54 STEP 2261 H=2261:00 +CPU= 27.816
  15:36:37 STEP 2262 H=2262:00 +CPU= 42.975
  15:37:05 STEP 2263 H=2263:00 +CPU= 27.833
  15:37:34 STEP 2264 H=2264:00 +CPU= 28.719
  15:38:16 STEP 2265 H=2265:00 +CPU= 42.311
  15:38:45 STEP 2266 H=2266:00 +CPU= 28.599
  15:39:13 STEP 2267 H=2267:00 +CPU= 27.831
  15:39:59 STEP 2268 H=2268:00 +CPU= 46.296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002268
  15:40:26 STEP 2269 H=2269:00 +CPU= 27.061
  15:40:54 STEP 2270 H=2270:00 +CPU= 27.751
  15:41:35 STEP 2271 H=2271:00 +CPU= 41.569
  15:42:03 STEP 2272 H=2272:00 +CPU= 27.841
  15:42:30 STEP 2273 H=2273:00 +CPU= 26.736
  15:43:12 STEP 2274 H=2274:00 +CPU= 42.394
  15:43:40 STEP 2275 H=2275:00 +CPU= 27.061
  15:44:07 STEP 2276 H=2276:00 +CPU= 27.736
  15:44:49 STEP 2277 H=2277:00 +CPU= 41.968
  15:45:18 STEP 2278 H=2278:00 +CPU= 28.505
  15:45:47 STEP 2279 H=2279:00 +CPU= 28.785
  15:46:37 STEP 2280 H=2280:00 +CPU= 49.408
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_94.zip
Uploading the intermediate file: upload_file_94.zip
  15:47:05 STEP 2281 H=2281:00 +CPU= 27.565
  15:47:33 STEP 2282 H=2282:00 +CPU= 28.421
  15:48:15 STEP 2283 H=2283:00 +CPU= 42.292
  15:48:44 STEP 2284 H=2284:00 +CPU= 28.526
  15:49:12 STEP 2285 H=2285:00 +CPU= 27.656
  15:49:55 STEP 2286 H=2286:00 +CPU= 43.181
  15:50:23 STEP 2287 H=2287:00 +CPU= 27.796
  15:50:51 STEP 2288 H=2288:00 +CPU= 28.653
  15:51:34 STEP 2289 H=2289:00 +CPU= 42.130
  15:52:02 STEP 2290 H=2290:00 +CPU= 28.493
  15:52:30 STEP 2291 H=2291:00 +CPU= 27.820
  15:53:17 STEP 2292 H=2292:00 +CPU= 46.897
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002292
  15:53:45 STEP 2293 H=2293:00 +CPU= 27.685
  15:54:13 STEP 2294 H=2294:00 +CPU= 28.554
  15:54:56 STEP 2295 H=2295:00 +CPU= 42.331
  15:55:24 STEP 2296 H=2296:00 +CPU= 28.570
  15:55:52 STEP 2297 H=2297:00 +CPU= 27.839
  15:56:36 STEP 2298 H=2298:00 +CPU= 43.493
  15:57:03 STEP 2299 H=2299:00 +CPU= 27.734
  15:57:32 STEP 2300 H=2300:00 +CPU= 28.493
  15:58:14 STEP 2301 H=2301:00 +CPU= 42.423
  15:58:43 STEP 2302 H=2302:00 +CPU= 28.472
  15:59:11 STEP 2303 H=2303:00 +CPU= 28.427
  16:00:01 STEP 2304 H=2304:00 +CPU= 48.648
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_95.zip
Uploading the intermediate file: upload_file_95.zip
  16:00:28 STEP 2305 H=2305:00 +CPU= 26.723
  16:00:55 STEP 2306 H=2306:00 +CPU= 26.697
  16:01:35 STEP 2307 H=2307:00 +CPU= 40.172
  16:02:02 STEP 2308 H=2308:00 +CPU= 26.986
  16:02:28 STEP 2309 H=2309:00 +CPU= 26.176
  16:03:09 STEP 2310 H=2310:00 +CPU= 40.812
  16:03:35 STEP 2311 H=2311:00 +CPU= 25.989
  16:04:02 STEP 2312 H=2312:00 +CPU= 26.669
  16:04:42 STEP 2313 H=2313:00 +CPU= 40.164
  16:05:09 STEP 2314 H=2314:00 +CPU= 26.932
  16:05:35 STEP 2315 H=2315:00 +CPU= 26.191
  16:06:20 STEP 2316 H=2316:00 +CPU= 44.331
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002316
  16:06:46 STEP 2317 H=2317:00 +CPU= 26.313
  16:07:13 STEP 2318 H=2318:00 +CPU= 26.699
  16:07:53 STEP 2319 H=2319:00 +CPU= 39.726
  16:08:19 STEP 2320 H=2320:00 +CPU= 26.800
  16:08:45 STEP 2321 H=2321:00 +CPU= 26.043
  16:09:26 STEP 2322 H=2322:00 +CPU= 40.906
  16:09:53 STEP 2323 H=2323:00 +CPU= 26.271
  16:10:19 STEP 2324 H=2324:00 +CPU= 26.672
  16:10:59 STEP 2325 H=2325:00 +CPU= 39.844
  16:11:26 STEP 2326 H=2326:00 +CPU= 26.473
  16:11:53 STEP 2327 H=2327:00 +CPU= 26.974
  16:12:40 STEP 2328 H=2328:00 +CPU= 46.865
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_96.zip
Uploading the intermediate file: upload_file_96.zip
  16:13:08 STEP 2329 H=2329:00 +CPU= 27.447
  16:13:35 STEP 2330 H=2330:00 +CPU= 27.344
  16:14:17 STEP 2331 H=2331:00 +CPU= 41.159
  16:14:44 STEP 2332 H=2332:00 +CPU= 27.489
  16:15:11 STEP 2333 H=2333:00 +CPU= 26.560
  16:15:52 STEP 2334 H=2334:00 +CPU= 41.671
  16:16:19 STEP 2335 H=2335:00 +CPU= 26.564
  16:16:47 STEP 2336 H=2336:00 +CPU= 27.801
  16:17:27 STEP 2337 H=2337:00 +CPU= 40.548
  16:17:55 STEP 2338 H=2338:00 +CPU= 27.672
  16:18:22 STEP 2339 H=2339:00 +CPU= 26.717
  16:19:07 STEP 2340 H=2340:00 +CPU= 44.804
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002340
  16:19:34 STEP 2341 H=2341:00 +CPU= 26.786
  16:20:01 STEP 2342 H=2342:00 +CPU= 27.429
  16:20:42 STEP 2343 H=2343:00 +CPU= 40.817
  16:21:09 STEP 2344 H=2344:00 +CPU= 27.384
  16:21:36 STEP 2345 H=2345:00 +CPU= 26.524
  16:22:17 STEP 2346 H=2346:00 +CPU= 41.579
  16:22:44 STEP 2347 H=2347:00 +CPU= 26.490
  16:23:11 STEP 2348 H=2348:00 +CPU= 27.412
  16:23:52 STEP 2349 H=2349:00 +CPU= 40.661
  16:24:20 STEP 2350 H=2350:00 +CPU= 27.452
  16:24:47 STEP 2351 H=2351:00 +CPU= 27.593
  16:25:35 STEP 2352 H=2352:00 +CPU= 47.042
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_97.zip
Uploading the intermediate file: upload_file_97.zip
  16:26:02 STEP 2353 H=2353:00 +CPU= 26.647
  16:26:30 STEP 2354 H=2354:00 +CPU= 27.615
  16:27:11 STEP 2355 H=2355:00 +CPU= 41.613
  16:27:39 STEP 2356 H=2356:00 +CPU= 27.625
  16:28:06 STEP 2357 H=2357:00 +CPU= 26.989
  16:28:49 STEP 2358 H=2358:00 +CPU= 42.713
  16:29:16 STEP 2359 H=2359:00 +CPU= 26.883
  16:29:44 STEP 2360 H=2360:00 +CPU= 27.867
  16:30:25 STEP 2361 H=2361:00 +CPU= 41.688
  16:30:53 STEP 2362 H=2362:00 +CPU= 27.709
  16:31:20 STEP 2363 H=2363:00 +CPU= 27.023
  16:32:06 STEP 2364 H=2364:00 +CPU= 46.015
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002364
  16:32:33 STEP 2365 H=2365:00 +CPU= 26.976
  16:33:01 STEP 2366 H=2366:00 +CPU= 27.996
  16:33:43 STEP 2367 H=2367:00 +CPU= 41.676
  16:34:11 STEP 2368 H=2368:00 +CPU= 27.878
  16:34:38 STEP 2369 H=2369:00 +CPU= 27.108
  16:35:21 STEP 2370 H=2370:00 +CPU= 42.767
  16:35:47 STEP 2371 H=2371:00 +CPU= 26.625
  16:36:15 STEP 2372 H=2372:00 +CPU= 28.045
  16:36:57 STEP 2373 H=2373:00 +CPU= 41.969
  16:37:25 STEP 2374 H=2374:00 +CPU= 27.949
  16:37:53 STEP 2375 H=2375:00 +CPU= 28.106
  16:38:43 STEP 2376 H=2376:00 +CPU= 48.823
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_98.zip
Uploading the intermediate file: upload_file_98.zip
  16:39:10 STEP 2377 H=2377:00 +CPU= 26.746
  16:39:38 STEP 2378 H=2378:00 +CPU= 27.895
  16:40:20 STEP 2379 H=2379:00 +CPU= 41.686
  16:40:48 STEP 2380 H=2380:00 +CPU= 27.771
  16:41:15 STEP 2381 H=2381:00 +CPU= 27.002
  16:41:58 STEP 2382 H=2382:00 +CPU= 42.599
  16:42:25 STEP 2383 H=2383:00 +CPU= 26.977
  16:42:52 STEP 2384 H=2384:00 +CPU= 27.827
  16:43:34 STEP 2385 H=2385:00 +CPU= 41.648
  16:44:02 STEP 2386 H=2386:00 +CPU= 27.697
  16:44:29 STEP 2387 H=2387:00 +CPU= 26.916
  16:45:15 STEP 2388 H=2388:00 +CPU= 46.070
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002388
  16:45:42 STEP 2389 H=2389:00 +CPU= 27.288
  16:46:10 STEP 2390 H=2390:00 +CPU= 28.031
  16:46:52 STEP 2391 H=2391:00 +CPU= 41.603
  16:47:20 STEP 2392 H=2392:00 +CPU= 27.959
  16:47:47 STEP 2393 H=2393:00 +CPU= 27.111
  16:48:30 STEP 2394 H=2394:00 +CPU= 42.619
  16:48:58 STEP 2395 H=2395:00 +CPU= 27.477
  16:49:26 STEP 2396 H=2396:00 +CPU= 27.967
  16:50:08 STEP 2397 H=2397:00 +CPU= 41.952
  16:50:36 STEP 2398 H=2398:00 +CPU= 28.164
  16:51:04 STEP 2399 H=2399:00 +CPU= 28.231
  16:51:55 STEP 2400 H=2400:00 +CPU= 48.778
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_99.zip
Uploading the intermediate file: upload_file_99.zip
  16:52:22 STEP 2401 H=2401:00 +CPU= 27.245
  16:52:50 STEP 2402 H=2402:00 +CPU= 28.127
  16:53:32 STEP 2403 H=2403:00 +CPU= 41.767
  16:54:00 STEP 2404 H=2404:00 +CPU= 28.162
  16:54:27 STEP 2405 H=2405:00 +CPU= 27.390
  16:55:10 STEP 2406 H=2406:00 +CPU= 42.822
  16:55:37 STEP 2407 H=2407:00 +CPU= 26.896
  16:56:05 STEP 2408 H=2408:00 +CPU= 27.729
  16:56:47 STEP 2409 H=2409:00 +CPU= 41.855
  16:57:15 STEP 2410 H=2410:00 +CPU= 27.697
  16:57:42 STEP 2411 H=2411:00 +CPU= 27.108
  16:58:28 STEP 2412 H=2412:00 +CPU= 46.385
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002412
  16:58:55 STEP 2413 H=2413:00 +CPU= 27.197
  16:59:23 STEP 2414 H=2414:00 +CPU= 28.047
  17:00:05 STEP 2415 H=2415:00 +CPU= 41.923
  17:00:33 STEP 2416 H=2416:00 +CPU= 27.892
  17:01:00 STEP 2417 H=2417:00 +CPU= 27.033
  17:01:43 STEP 2418 H=2418:00 +CPU= 42.911
  17:02:10 STEP 2419 H=2419:00 +CPU= 26.872
  17:02:38 STEP 2420 H=2420:00 +CPU= 27.754
  17:03:19 STEP 2421 H=2421:00 +CPU= 41.392
  17:03:47 STEP 2422 H=2422:00 +CPU= 27.675
  17:04:15 STEP 2423 H=2423:00 +CPU= 27.973
  17:05:05 STEP 2424 H=2424:00 +CPU= 49.216
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_100.zip
Uploading the intermediate file: upload_file_100.zip
  17:05:32 STEP 2425 H=2425:00 +CPU= 26.841
  17:06:00 STEP 2426 H=2426:00 +CPU= 27.877
  17:06:42 STEP 2427 H=2427:00 +CPU= 41.767
  17:07:10 STEP 2428 H=2428:00 +CPU= 27.930
  17:07:37 STEP 2429 H=2429:00 +CPU= 27.181
  17:08:20 STEP 2430 H=2430:00 +CPU= 42.481
  17:08:47 STEP 2431 H=2431:00 +CPU= 27.248
  17:09:15 STEP 2432 H=2432:00 +CPU= 27.982
  17:09:57 STEP 2433 H=2433:00 +CPU= 41.380
  17:10:25 STEP 2434 H=2434:00 +CPU= 28.150
  17:10:52 STEP 2435 H=2435:00 +CPU= 27.092
  17:11:38 STEP 2436 H=2436:00 +CPU= 46.376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002436
  17:12:05 STEP 2437 H=2437:00 +CPU= 26.859
  17:12:35 STEP 2438 H=2438:00 +CPU= 28.107
  17:13:16 STEP 2439 H=2439:00 +CPU= 40.879
  17:13:43 STEP 2440 H=2440:00 +CPU= 27.494
  17:14:10 STEP 2441 H=2441:00 +CPU= 26.774
  17:14:52 STEP 2442 H=2442:00 +CPU= 41.831
  17:15:19 STEP 2443 H=2443:00 +CPU= 26.790
  17:15:46 STEP 2444 H=2444:00 +CPU= 27.537
  17:16:27 STEP 2445 H=2445:00 +CPU= 40.010
  17:16:54 STEP 2446 H=2446:00 +CPU= 27.090
  17:17:20 STEP 2447 H=2447:00 +CPU= 26.781
  17:18:09 STEP 2448 H=2448:00 +CPU= 46.640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_101.zip
Uploading the intermediate file: upload_file_101.zip
  17:18:36 STEP 2449 H=2449:00 +CPU= 27.571
  17:19:05 STEP 2450 H=2450:00 +CPU= 28.753
  17:19:47 STEP 2451 H=2451:00 +CPU= 41.929
  17:20:15 STEP 2452 H=2452:00 +CPU= 28.301
  17:20:43 STEP 2453 H=2453:00 +CPU= 27.210
  17:21:25 STEP 2454 H=2454:00 +CPU= 42.668
  17:21:53 STEP 2455 H=2455:00 +CPU= 27.235
  17:22:21 STEP 2456 H=2456:00 +CPU= 28.119
  17:23:02 STEP 2457 H=2457:00 +CPU= 41.533
  17:23:30 STEP 2458 H=2458:00 +CPU= 28.003
  17:23:58 STEP 2459 H=2459:00 +CPU= 27.312
  17:24:44 STEP 2460 H=2460:00 +CPU= 46.502
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002460
  17:25:11 STEP 2461 H=2461:00 +CPU= 27.252
  17:25:39 STEP 2462 H=2462:00 +CPU= 28.024
  17:26:21 STEP 2463 H=2463:00 +CPU= 41.622
  17:26:49 STEP 2464 H=2464:00 +CPU= 28.057
  17:27:16 STEP 2465 H=2465:00 +CPU= 27.214
  17:27:59 STEP 2466 H=2466:00 +CPU= 42.407
  17:28:26 STEP 2467 H=2467:00 +CPU= 27.265
  17:28:54 STEP 2468 H=2468:00 +CPU= 27.978
  17:29:36 STEP 2469 H=2469:00 +CPU= 41.478
  17:30:04 STEP 2470 H=2470:00 +CPU= 28.071
  17:30:32 STEP 2471 H=2471:00 +CPU= 28.085
  17:31:20 STEP 2472 H=2472:00 +CPU= 47.544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_102.zip
Uploading the intermediate file: upload_file_102.zip
Uploading trickle at timestep: 8899200
  17:31:48 STEP 2473 H=2473:00 +CPU= 26.817
  17:32:15 STEP 2474 H=2474:00 +CPU= 27.603
  17:32:57 STEP 2475 H=2475:00 +CPU= 41.349
  17:33:24 STEP 2476 H=2476:00 +CPU= 27.651
  17:33:51 STEP 2477 H=2477:00 +CPU= 26.700
  17:34:34 STEP 2478 H=2478:00 +CPU= 42.707
  17:35:01 STEP 2479 H=2479:00 +CPU= 26.799
  17:35:29 STEP 2480 H=2480:00 +CPU= 28.130
  17:36:10 STEP 2481 H=2481:00 +CPU= 41.648
  17:36:39 STEP 2482 H=2482:00 +CPU= 28.044
  17:37:05 STEP 2483 H=2483:00 +CPU= 26.899
  17:37:52 STEP 2484 H=2484:00 +CPU= 46.179
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002484
  17:38:19 STEP 2485 H=2485:00 +CPU= 27.015
  17:38:47 STEP 2486 H=2486:00 +CPU= 27.933
  17:39:28 STEP 2487 H=2487:00 +CPU= 41.378
  17:39:56 STEP 2488 H=2488:00 +CPU= 27.735
  17:40:22 STEP 2489 H=2489:00 +CPU= 26.598
  17:41:05 STEP 2490 H=2490:00 +CPU= 42.157
  17:41:31 STEP 2491 H=2491:00 +CPU= 26.709
  17:41:59 STEP 2492 H=2492:00 +CPU= 27.643
  17:42:40 STEP 2493 H=2493:00 +CPU= 41.436
  17:43:08 STEP 2494 H=2494:00 +CPU= 27.672
  17:43:36 STEP 2495 H=2495:00 +CPU= 27.944
  17:44:26 STEP 2496 H=2496:00 +CPU= 48.322
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_103.zip
Uploading the intermediate file: upload_file_103.zip
  17:44:52 STEP 2497 H=2497:00 +CPU= 26.626
  17:45:20 STEP 2498 H=2498:00 +CPU= 27.756
  17:46:01 STEP 2499 H=2499:00 +CPU= 41.115
  17:46:29 STEP 2500 H=2500:00 +CPU= 27.622
  17:46:56 STEP 2501 H=2501:00 +CPU= 26.810
  17:47:38 STEP 2502 H=2502:00 +CPU= 42.542
  17:48:05 STEP 2503 H=2503:00 +CPU= 26.862
  17:48:33 STEP 2504 H=2504:00 +CPU= 28.025
  17:49:15 STEP 2505 H=2505:00 +CPU= 41.364
  17:49:43 STEP 2506 H=2506:00 +CPU= 27.981
  17:50:09 STEP 2507 H=2507:00 +CPU= 26.763
  17:50:55 STEP 2508 H=2508:00 +CPU= 45.953
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002508
  17:51:23 STEP 2509 H=2509:00 +CPU= 27.249
  17:51:50 STEP 2510 H=2510:00 +CPU= 27.767
  17:52:32 STEP 2511 H=2511:00 +CPU= 41.272
  17:52:59 STEP 2512 H=2512:00 +CPU= 27.728
  17:53:26 STEP 2513 H=2513:00 +CPU= 26.792
  17:54:09 STEP 2514 H=2514:00 +CPU= 42.394
  17:54:35 STEP 2515 H=2515:00 +CPU= 26.864
  17:55:03 STEP 2516 H=2516:00 +CPU= 27.578
  17:55:45 STEP 2517 H=2517:00 +CPU= 41.557
  17:56:12 STEP 2518 H=2518:00 +CPU= 27.595
  17:56:40 STEP 2519 H=2519:00 +CPU= 28.132
  17:57:30 STEP 2520 H=2520:00 +CPU= 48.434
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_104.zip
Uploading the intermediate file: upload_file_104.zip
  17:57:57 STEP 2521 H=2521:00 +CPU= 27.133
  17:58:25 STEP 2522 H=2522:00 +CPU= 27.710
  17:59:06 STEP 2523 H=2523:00 +CPU= 41.237
  17:59:34 STEP 2524 H=2524:00 +CPU= 27.917
  18:00:02 STEP 2525 H=2525:00 +CPU= 27.597
  18:00:45 STEP 2526 H=2526:00 +CPU= 42.961
  18:01:13 STEP 2527 H=2527:00 +CPU= 27.833
  18:01:41 STEP 2528 H=2528:00 +CPU= 28.478
  18:02:23 STEP 2529 H=2529:00 +CPU= 42.018
  18:02:52 STEP 2530 H=2530:00 +CPU= 28.497
  18:03:19 STEP 2531 H=2531:00 +CPU= 27.498
  18:04:06 STEP 2532 H=2532:00 +CPU= 46.833
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002532
  18:04:34 STEP 2533 H=2533:00 +CPU= 27.776
  18:05:03 STEP 2534 H=2534:00 +CPU= 28.969
  18:05:46 STEP 2535 H=2535:00 +CPU= 42.556
  18:06:14 STEP 2536 H=2536:00 +CPU= 28.705
  18:06:42 STEP 2537 H=2537:00 +CPU= 27.870
  18:07:26 STEP 2538 H=2538:00 +CPU= 43.391
  18:07:54 STEP 2539 H=2539:00 +CPU= 27.934
  18:08:23 STEP 2540 H=2540:00 +CPU= 28.657
  18:09:05 STEP 2541 H=2541:00 +CPU= 42.435
  18:09:34 STEP 2542 H=2542:00 +CPU= 28.603
  18:10:02 STEP 2543 H=2543:00 +CPU= 28.601
  18:10:54 STEP 2544 H=2544:00 +CPU= 49.625
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:11:22 STEP 2545 H=2545:00 +CPU= 27.188
  18:11:49 STEP 2546 H=2546:00 +CPU= 27.393
  18:12:30 STEP 2547 H=2547:00 +CPU= 41.128
  18:12:58 STEP 2548 H=2548:00 +CPU= 27.677
  18:13:25 STEP 2549 H=2549:00 +CPU= 26.575
  18:14:07 STEP 2550 H=2550:00 +CPU= 41.902
  18:14:33 STEP 2551 H=2551:00 +CPU= 26.648
  18:15:01 STEP 2552 H=2552:00 +CPU= 27.601
  18:15:42 STEP 2553 H=2553:00 +CPU= 41.097
  18:16:10 STEP 2554 H=2554:00 +CPU= 27.372
  18:16:37 STEP 2555 H=2555:00 +CPU= 26.852
  18:17:23 STEP 2556 H=2556:00 +CPU= 45.972
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002556
  18:17:50 STEP 2557 H=2557:00 +CPU= 27.229
  18:18:18 STEP 2558 H=2558:00 +CPU= 27.976
  18:18:59 STEP 2559 H=2559:00 +CPU= 41.170
  18:19:27 STEP 2560 H=2560:00 +CPU= 28.154
  18:19:55 STEP 2561 H=2561:00 +CPU= 27.259
  18:20:38 STEP 2562 H=2562:00 +CPU= 43.144
  18:21:06 STEP 2563 H=2563:00 +CPU= 27.713
  18:21:35 STEP 2564 H=2564:00 +CPU= 28.613
  18:22:17 STEP 2565 H=2565:00 +CPU= 42.145
  18:22:45 STEP 2566 H=2566:00 +CPU= 28.542
  18:23:14 STEP 2567 H=2567:00 +CPU= 28.460
  18:24:05 STEP 2568 H=2568:00 +CPU= 49.722
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:24:32 STEP 2569 H=2569:00 +CPU= 27.534
  18:25:01 STEP 2570 H=2570:00 +CPU= 28.617
  18:25:44 STEP 2571 H=2571:00 +CPU= 42.430
  18:26:12 STEP 2572 H=2572:00 +CPU= 28.195
  18:26:40 STEP 2573 H=2573:00 +CPU= 27.582
  18:27:23 STEP 2574 H=2574:00 +CPU= 43.407
  18:27:51 STEP 2575 H=2575:00 +CPU= 27.788
  18:28:20 STEP 2576 H=2576:00 +CPU= 28.614
  18:29:02 STEP 2577 H=2577:00 +CPU= 42.563
  18:29:31 STEP 2578 H=2578:00 +CPU= 28.459
  18:29:58 STEP 2579 H=2579:00 +CPU= 27.607
  18:30:45 STEP 2580 H=2580:00 +CPU= 46.870
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002580
  18:31:13 STEP 2581 H=2581:00 +CPU= 27.690
  18:31:42 STEP 2582 H=2582:00 +CPU= 28.493
  18:32:24 STEP 2583 H=2583:00 +CPU= 42.157
  18:32:52 STEP 2584 H=2584:00 +CPU= 28.481
  18:33:20 STEP 2585 H=2585:00 +CPU= 27.749
  18:34:03 STEP 2586 H=2586:00 +CPU= 43.256
  18:34:31 STEP 2587 H=2587:00 +CPU= 27.329
  18:34:59 STEP 2588 H=2588:00 +CPU= 28.489
  18:35:42 STEP 2589 H=2589:00 +CPU= 42.268
  18:36:10 STEP 2590 H=2590:00 +CPU= 28.277
  18:36:38 STEP 2591 H=2591:00 +CPU= 28.447
  18:37:29 STEP 2592 H=2592:00 +CPU= 49.353
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:37:56 STEP 2593 H=2593:00 +CPU= 27.542
  18:38:25 STEP 2594 H=2594:00 +CPU= 28.078
  18:39:06 STEP 2595 H=2595:00 +CPU= 41.454
  18:39:34 STEP 2596 H=2596:00 +CPU= 28.005
  18:40:01 STEP 2597 H=2597:00 +CPU= 26.856
  18:40:44 STEP 2598 H=2598:00 +CPU= 42.487
  18:41:11 STEP 2599 H=2599:00 +CPU= 26.887
  18:41:38 STEP 2600 H=2600:00 +CPU= 27.801
  18:42:20 STEP 2601 H=2601:00 +CPU= 41.503
  18:42:48 STEP 2602 H=2602:00 +CPU= 27.803
  18:43:15 STEP 2603 H=2603:00 +CPU= 26.955
  18:44:01 STEP 2604 H=2604:00 +CPU= 46.116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
  18:44:28 STEP 2605 H=2605:00 +CPU= 26.948
  18:44:56 STEP 2606 H=2606:00 +CPU= 27.818
  18:45:37 STEP 2607 H=2607:00 +CPU= 41.485
  18:46:05 STEP 2608 H=2608:00 +CPU= 27.772
  18:46:32 STEP 2609 H=2609:00 +CPU= 26.861
  18:47:15 STEP 2610 H=2610:00 +CPU= 42.766
  18:47:41 STEP 2611 H=2611:00 +CPU= 26.768
  18:48:09 STEP 2612 H=2612:00 +CPU= 28.050
  18:48:51 STEP 2613 H=2613:00 +CPU= 41.633
  18:49:19 STEP 2614 H=2614:00 +CPU= 27.894
  18:49:47 STEP 2615 H=2615:00 +CPU= 28.072
  18:50:37 STEP 2616 H=2616:00 +CPU= 48.617
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_108.zip
Uploading the intermediate file: upload_file_108.zip
  18:51:05 STEP 2617 H=2617:00 +CPU= 27.700
  18:51:33 STEP 2618 H=2618:00 +CPU= 28.542
  18:52:16 STEP 2619 H=2619:00 +CPU= 42.291
  18:52:44 STEP 2620 H=2620:00 +CPU= 28.714
  18:53:12 STEP 2621 H=2621:00 +CPU= 27.965
  18:53:56 STEP 2622 H=2622:00 +CPU= 43.362
  18:54:24 STEP 2623 H=2623:00 +CPU= 28.197
  18:54:53 STEP 2624 H=2624:00 +CPU= 29.103
  18:55:36 STEP 2625 H=2625:00 +CPU= 42.547
  18:56:04 STEP 2626 H=2626:00 +CPU= 28.673
  18:56:32 STEP 2627 H=2627:00 +CPU= 27.920
  18:57:19 STEP 2628 H=2628:00 +CPU= 46.495
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
  18:57:47 STEP 2629 H=2629:00 +CPU= 27.705
  18:58:15 STEP 2630 H=2630:00 +CPU= 28.378
  18:58:57 STEP 2631 H=2631:00 +CPU= 42.036
  18:59:25 STEP 2632 H=2632:00 +CPU= 28.379
  18:59:53 STEP 2633 H=2633:00 +CPU= 27.427
  19:00:36 STEP 2634 H=2634:00 +CPU= 42.796
  19:01:03 STEP 2635 H=2635:00 +CPU= 27.331
  19:01:31 STEP 2636 H=2636:00 +CPU= 28.085
  19:02:13 STEP 2637 H=2637:00 +CPU= 41.842
  19:02:41 STEP 2638 H=2638:00 +CPU= 28.209
  19:03:09 STEP 2639 H=2639:00 +CPU= 28.285
  19:04:00 STEP 2640 H=2640:00 +CPU= 49.085
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_109.zip
Uploading the intermediate file: upload_file_109.zip
  19:04:27 STEP 2641 H=2641:00 +CPU= 26.677
  19:04:55 STEP 2642 H=2642:00 +CPU= 28.021
  19:05:36 STEP 2643 H=2643:00 +CPU= 41.602
  19:06:05 STEP 2644 H=2644:00 +CPU= 28.080
  19:06:31 STEP 2645 H=2645:00 +CPU= 26.856
  19:07:14 STEP 2646 H=2646:00 +CPU= 42.331
  19:07:41 STEP 2647 H=2647:00 +CPU= 26.886
  19:08:08 STEP 2648 H=2648:00 +CPU= 27.698
  19:08:50 STEP 2649 H=2649:00 +CPU= 41.206
  19:09:17 STEP 2650 H=2650:00 +CPU= 27.721
  19:09:44 STEP 2651 H=2651:00 +CPU= 26.871
  19:10:30 STEP 2652 H=2652:00 +CPU= 46.127
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652
  19:10:57 STEP 2653 H=2653:00 +CPU= 26.778
  19:11:25 STEP 2654 H=2654:00 +CPU= 27.631
  19:12:07 STEP 2655 H=2655:00 +CPU= 41.673
  19:12:36 STEP 2656 H=2656:00 +CPU= 27.613
  19:13:03 STEP 2657 H=2657:00 +CPU= 26.994
  19:13:45 STEP 2658 H=2658:00 +CPU= 42.049
  19:14:12 STEP 2659 H=2659:00 +CPU= 26.927
  19:14:39 STEP 2660 H=2660:00 +CPU= 27.395
  19:15:20 STEP 2661 H=2661:00 +CPU= 40.815
  19:15:47 STEP 2662 H=2662:00 +CPU= 27.363
  19:16:15 STEP 2663 H=2663:00 +CPU= 27.622
  19:17:04 STEP 2664 H=2664:00 +CPU= 47.410
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_110.zip
Uploading the intermediate file: upload_file_110.zip
  19:17:31 STEP 2665 H=2665:00 +CPU= 26.776
  19:17:59 STEP 2666 H=2666:00 +CPU= 28.162
  19:18:41 STEP 2667 H=2667:00 +CPU= 42.025
  19:19:09 STEP 2668 H=2668:00 +CPU= 27.999
  19:19:36 STEP 2669 H=2669:00 +CPU= 26.955
  19:20:19 STEP 2670 H=2670:00 +CPU= 42.487
  19:20:46 STEP 2671 H=2671:00 +CPU= 26.793
  19:21:13 STEP 2672 H=2672:00 +CPU= 27.740
  19:21:55 STEP 2673 H=2673:00 +CPU= 41.354
  19:22:21 STEP 2674 H=2674:00 +CPU= 26.499
  19:22:47 STEP 2675 H=2675:00 +CPU= 25.611
  19:23:31 STEP 2676 H=2676:00 +CPU= 44.155
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676
  19:23:57 STEP 2677 H=2677:00 +CPU= 25.475
  19:24:23 STEP 2678 H=2678:00 +CPU= 26.622
  19:25:03 STEP 2679 H=2679:00 +CPU= 39.782
  19:25:30 STEP 2680 H=2680:00 +CPU= 26.752
  19:25:56 STEP 2681 H=2681:00 +CPU= 25.580
  19:26:36 STEP 2682 H=2682:00 +CPU= 40.516
  19:27:02 STEP 2683 H=2683:00 +CPU= 25.556
  19:27:29 STEP 2684 H=2684:00 +CPU= 26.860
  19:28:08 STEP 2685 H=2685:00 +CPU= 39.615
  19:28:35 STEP 2686 H=2686:00 +CPU= 26.577
  19:29:01 STEP 2687 H=2687:00 +CPU= 26.477
  19:29:49 STEP 2688 H=2688:00 +CPU= 46.605
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_111.zip
Uploading the intermediate file: upload_file_111.zip
  19:30:16 STEP 2689 H=2689:00 +CPU= 26.868
  19:30:44 STEP 2690 H=2690:00 +CPU= 27.735
  19:31:26 STEP 2691 H=2691:00 +CPU= 41.622
  19:31:54 STEP 2692 H=2692:00 +CPU= 27.892
  19:32:21 STEP 2693 H=2693:00 +CPU= 26.845
  19:33:04 STEP 2694 H=2694:00 +CPU= 42.940
  19:33:31 STEP 2695 H=2695:00 +CPU= 26.997
  19:33:59 STEP 2696 H=2696:00 +CPU= 28.148
  19:34:41 STEP 2697 H=2697:00 +CPU= 41.723
  19:35:09 STEP 2698 H=2698:00 +CPU= 28.271
  19:35:36 STEP 2699 H=2699:00 +CPU= 27.328
  19:36:23 STEP 2700 H=2700:00 +CPU= 46.290
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700
  19:36:50 STEP 2701 H=2701:00 +CPU= 27.141
  19:37:18 STEP 2702 H=2702:00 +CPU= 28.167
  19:38:00 STEP 2703 H=2703:00 +CPU= 41.590
  19:38:28 STEP 2704 H=2704:00 +CPU= 28.035
  19:38:55 STEP 2705 H=2705:00 +CPU= 26.952
  19:39:37 STEP 2706 H=2706:00 +CPU= 42.475
  19:40:04 STEP 2707 H=2707:00 +CPU= 26.863
  19:40:32 STEP 2708 H=2708:00 +CPU= 27.694
  19:41:14 STEP 2709 H=2709:00 +CPU= 41.760
  19:41:41 STEP 2710 H=2710:00 +CPU= 27.623
  19:42:09 STEP 2711 H=2711:00 +CPU= 27.929
  19:42:59 STEP 2712 H=2712:00 +CPU= 48.774
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824_112.zip
Uploading the intermediate file: upload_file_112.zip
Uploading trickle at timestep: 9763200
  19:43:26 STEP 2713 H=2713:00 +CPU= 26.267
  19:43:53 STEP 2714 H=2714:00 +CPU= 27.633
  19:44:34 STEP 2715 H=2715:00 +CPU= 40.653
  19:45:01 STEP 2716 H=2716:00 +CPU= 27.081
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0014
(argv4) batchid: 981
(argv5) wuid: 12197658
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12197658
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fd9eb4767799b515337534a2ffac3145 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0014_2012050100_123_981_12197658.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0014_2012050100_123_981_12197658.zip
ic_ancil_file: ic_ancil_12197658
ifsdata_file: ifsdata_12197658
climate_data_file: clim_data_12197658
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_26b94e10c63f9cc0e4106fd553b9c042 to: /var/lib/boinc-client/slots/6/ic_ancil_12197658.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12197658.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9cf02900c6987fd00d7d2f5cda2f7617 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12197658.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12197658.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12197658.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12197658.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12197658.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12197658.xml
-- Model is restarting --
Adjusting last_iter, 2717, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12197658.xml
last_cpu_time: 81747
upload_file_number: 113
last_iter: 2711
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0014_2012050100_123_981_12197658_1_r2123544824
The child process has been launched with process id: 17492
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ken-Z230-2:1:1:17492:17492] [20230101:194627:1672631187:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:47:27 STEP 2712 H=2712:00 +CPU= 48.163
forrtl: severe (41): insufficient virtual memory
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001D2B9CB  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000D78293  call_sl_heap_              92  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:05:39 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 9,763,200 81,616 0.0084
09 May 2023 16:05:34 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 8,899,200 73,744 0.0083
09 May 2023 16:05:25 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 7,776,000 63,630 0.0082
09 May 2023 16:05:21 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 6,912,000 56,904 0.0082
09 May 2023 16:05:16 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 6,048,000 50,169 0.0083
09 May 2023 16:05:12 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 5,184,000 43,433 0.0084
09 May 2023 16:05:07 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 4,320,000 36,192 0.0084
09 May 2023 16:05:02 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 3,456,000 28,857 0.0083
09 May 2023 16:04:58 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 2,592,000 21,518 0.0083
09 May 2023 16:04:53 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 1,728,000 14,230 0.0082
09 May 2023 16:04:48 1515225 22294856 oifs_43r3_ps_0014_2012050100_123_981_12197658_1 864,000 7,031 0.0081


©2024 climateprediction.net