Task 15802561

Name	hadcm3n_4fwv_1940_40_008302790_4
Workunit	8453925
Created	29 May 2013, 20:33:39 UTC
Sent	20 Jun 2013, 9:33:28 UTC
Report deadline	19 Sep 2013, 17:00:39 UTC
Received	10 Jul 2013, 7:02:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1215740
Run time	13 days 17 hours 1 min 41 sec
CPU time	10 days 9 hours 2 min 58 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	2.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Ocean Restart file copy failed on 4fwvko.dae0c60 Ocean Restart file copy failed on 4fwvko.dae0c70 Ocean Restart file copy failed on 4fwvko.dae0c80 Ocean Restart file copy failed on 4fwvko.dae0c90 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae0ce0 Ocean Restart file copy failed on 4fwvko.dae0cf0 Ocean Restart file copy failed on 4fwvko.dae0cg0 Ocean Restart file copy failed on 4fwvko.dae0ch0 Ocean Restart file copy failed on 4fwvko.dae0co0 Ocean Restart file copy failed on 4fwvko.dae0cp0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 4fwvko.dae1160 Ocean Restart file copy failed on 4fwvko.dae1170 Ocean Restart file copy failed on 4fwvko.dae11b0 Ocean Restart file copy failed on 4fwvko.dae11c0 Ocean Restart file copy failed on 4fwvko.dae11d0 Ocean Restart file copy failed on 4fwvko.dae11e0 Ocean Restart file copy failed on 4fwvko.dae11f0 Ocean Restart file copy failed on 4fwvko.dae11g0 Ocean Restart file copy failed on 4fwvko.dae11h0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae14d0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae1680 Ocean Restart file copy failed on 4fwvko.dae1690 Ocean Restart file copy failed on 4fwvko.dae16a0 Ocean Restart file copy failed on 4fwvko.dae16b0 Ocean Restart file copy failed on 4fwvko.dae16c0 Ocean Restart file copy failed on 4fwvko.dae16d0 Ocean Restart file copy failed on 4fwvko.dae16e0 Ocean Restart file copy failed on 4fwvko.dae16f0 Ocean Restart file copy failed on 4fwvko.dae16g0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae17q0 Ocean Restart file copy failed on 4fwvko.dae17r0 Ocean Restart file copy failed on 4fwvko.dae17s0 Ocean Restart file copy failed on 4fwvko.dae17t0 Ocean Restart file copy failed on 4fwvko.dae17u0 Ocean Restart file copy failed on 4fwvko.dae1810 Ocean Restart file copy failed on 4fwvko.dae1820 Ocean Restart file copy failed on 4fwvko.dae1830 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1248, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae1aj0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae1ak0 Ocean Restart file copy failed on 4fwvko.dae1at0 Ocean Restart file copy failed on 4fwvko.dae1au0 Ocean Restart file copy failed on 4fwvko.dae1b10 Ocean Restart file copy failed on 4fwvko.dae1b20 Ocean Restart file copy failed on 4fwvko.dae1b30 Ocean Restart file copy failed on 4fwvko.dae1b40 Ocean Restart file copy failed on 4fwvko.dae1b50 Ocean Restart file copy failed on 4fwvko.dae1b60 Ocean Restart file copy failed on 4fwvko.dae1b70 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae1bm0 Ocean Restart file copy failed on 4fwvko.dae1bn0 Ocean Restart file copy failed on 4fwvko.dae1bo0 Ocean Restart file copy failed on 4fwvko.dae1bp0 Ocean Restart file copy failed on 4fwvko.dae1bq0 Ocean Restart file copy failed on 4fwvko.dae1br0 Ocean Restart file copy failed on 4fwvko.dae1bs0 Ocean Restart file copy failed on 4fwvko.dae1bt0 Ocean Restart file copy failed on 4fwvko.dae1bu0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae2390 Ocean Restart file copy failed on 4fwvko.dae23a0 Ocean Restart file copy failed on 4fwvko.dae23b0 Ocean Restart file copy failed on 4fwvko.dae23c0 Ocean Restart file copy failed on 4fwvko.dae23d0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 4fwvko.dae25b0 Ocean Restart file copy failed on 4fwvko.dae2990 Ocean Restart file copy failed on 4fwvko.dae29a0 Ocean Restart file copy failed on 4fwvko.dae29b0 Ocean Restart file copy failed on 4fwvko.dae29c0 Ocean Restart file copy failed on 4fwvko.dae29d0 Ocean Restart file copy failed on 4fwvko.dae29e0 Ocean Restart file copy failed on 4fwvko.dae29f0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae2a50 Ocean Restart file copy failed on 4fwvko.dae2a60 Ocean Restart file copy failed on 4fwvko.dae2a70 Ocean Restart file copy failed on 4fwvko.dae2a80 Ocean Restart file copy failed on 4fwvko.dae2a90 Ocean Restart file copy failed on 4fwvko.dae2aa0 Ocean Restart file copy failed on 4fwvko.dae2ab0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae2b50 Ocean Restart file copy failed on 4fwvko.dae2b60 Ocean Restart file copy failed on 4fwvko.dae2b70 Ocean Restart file copy failed on 4fwvko.dae2b80 Ocean Restart file copy failed on 4fwvko.dae2b90 Ocean Restart file copy failed on 4fwvko.dae2ba0 Ocean Restart file copy failed on 4fwvko.dae2bb0 Ocean Restart file copy failed on 4fwvko.dae2bc0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae2cu0 Ocean Restart file copy failed on 4fwvko.dae3110 Ocean Restart file copy failed on 4fwvko.dae3120 Ocean Restart file copy failed on 4fwvko.dae3130 Ocean Restart file copy failed on 4fwvko.dae3140 Ocean Restart file copy failed on 4fwvko.dae3150 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae35e0 Ocean Restart file copy failed on 4fwvko.dae35f0 Ocean Restart file copy failed on 4fwvko.dae35g0 Ocean Restart file copy failed on 4fwvko.dae35h0 Ocean Restart file copy failed on 4fwvko.dae35i0 Ocean Restart file copy failed on 4fwvko.dae35j0 Ocean Restart file copy failed on 4fwvko.dae35k0 Ocean Restart file copy failed on 4fwvko.dae35l0 Ocean Restart file copy failed on 4fwvko.dae35m0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae37u0 Ocean Restart file copy failed on 4fwvko.dae3810 Ocean Restart file copy failed on 4fwvko.dae3820 Ocean Restart file copy failed on 4fwvko.dae3830 Ocean Restart file copy failed on 4fwvko.dae3840 Ocean Restart file copy failed on 4fwvko.dae3850 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae39p0 Ocean Restart file copy failed on 4fwvko.dae39q0 Ocean Restart file copy failed on 4fwvko.dae39r0 Ocean Restart file copy failed on 4fwvko.dae39s0 Ocean Restart file copy failed on 4fwvko.dae39t0 Ocean Restart file copy failed on 4fwvko.dae39u0 Ocean Restart file copy failed on 4fwvko.dae3a10 Ocean Restart file copy failed on 4fwvko.dae3ag0 Ocean Restart file copy failed on 4fwvko.dae3ah0 Ocean Restart file copy failed on 4fwvko.dae3ai0 Ocean Restart file copy failed on 4fwvko.dae3aj0 Ocean Restart file copy failed on 4fwvko.dae3ak0 Ocean Restart file copy failed on 4fwvko.dae3al0 Ocean Restart file copy failed on 4fwvko.dae3am0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae3b10 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae3b20 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae3bl0 Ocean Restart file copy failed on 4fwvko.dae3bm0 Ocean Restart file copy failed on 4fwvko.dae3bn0 Ocean Restart file copy failed on 4fwvko.dae3bo0 Ocean Restart file copy failed on 4fwvko.dae3bp0 Ocean Restart file copy failed on 4fwvko.dae3bq0 Ocean Restart file copy failed on 4fwvko.dae3br0 Ocean Restart file copy failed on 4fwvko.dae3bs0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae3c40 Ocean Restart file copy failed on 4fwvko.dae3c50 Ocean Restart file copy failed on 4fwvko.dae3c60 Ocean Restart file copy failed on 4fwvko.dae3c70 Ocean Restart file copy failed on 4fwvko.dae3c80 Ocean Restart file copy failed on 4fwvko.dae3c90 Ocean Restart file copy failed on 4fwvko.dae3ca0 Ocean Restart file copy failed on 4fwvko.dae3cb0 Ocean Restart file copy failed on 4fwvko.dae3cc0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae4150 Ocean Restart file copy failed on 4fwvko.dae4160 Ocean Restart file copy failed on 4fwvko.dae4170 Ocean Restart file copy failed on 4fwvko.dae4180 Ocean Restart file copy failed on 4fwvko.dae4190 Ocean Restart file copy failed on 4fwvko.dae41a0 Ocean Restart file copy failed on 4fwvko.dae41b0 Ocean Restart file copy failed on 4fwvko.dae41c0 Ocean Restart file copy failed on 4fwvko.dae41d0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae4220 Ocean Restart file copy failed on 4fwvko.dae4230 Ocean Restart file copy failed on 4fwvko.dae4240 Ocean Restart file copy failed on 4fwvko.dae4250 Ocean Restart file copy failed on 4fwvko.dae4260 Ocean Restart file copy failed on 4fwvko.dae4270 Ocean Restart file copy failed on 4fwvko.dae4280 Ocean Restart file copy failed on 4fwvko.dae4290 Ocean Restart file copy failed on 4fwvko.dae42a0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae43q0 Ocean Restart file copy failed on 4fwvko.dae43r0 Ocean Restart file copy failed on 4fwvko.dae43s0 Ocean Restart file copy failed on 4fwvko.dae43t0 Ocean Restart file copy failed on 4fwvko.dae43u0 Ocean Restart file copy failed on 4fwvko.dae4410 Ocean Restart file copy failed on 4fwvko.dae4420 Ocean Restart file copy failed on 4fwvko.dae4430 Ocean Restart file copy failed on 4fwvko.dae4440 Ocean Restart file copy failed on 4fwvko.dae4450 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae45s0 Ocean Restart file copy failed on 4fwvko.dae45t0 Ocean Restart file copy failed on 4fwvko.dae45u0 Ocean Restart file copy failed on 4fwvko.dae4610 Ocean Restart file copy failed on 4fwvko.dae4620 Ocean Restart file copy failed on 4fwvko.dae4630 Ocean Restart file copy failed on 4fwvko.dae4640 Ocean Restart file copy failed on 4fwvko.dae4650 Ocean Restart file copy failed on 4fwvko.dae4660 Ocean Restart file copy failed on 4fwvko.dae4670 Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5560, iMonCtr=1 Model crash detected, will try to restart... COcean Restart file copy failed on 4fwvko.dae48m0 Ocean Restart file copy failed on 4fwvko.dae48n0 Ocean Restart file copy failed on 4fwvko.dae48o0 Ocean Restart file copy failed on 4fwvko.dae48p0 Ocean Restart file copy failed on 4fwvko.dae48q0 Ocean Restart file copy failed on 4fwvko.dae48r0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae49f0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae49g0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae49u0 Ocean Restart file copy failed on 4fwvko.dae4a10 Ocean Restart file copy failed on 4fwvko.dae4a20 Ocean Restart file copy failed on 4fwvko.dae4a30 Ocean Restart file copy failed on 4fwvko.dae4a40 Ocean Restart file copy failed on 4fwvko.dae4a50 Ocean Restart file copy failed on 4fwvko.dae4a60 Ocean Restart file copy failed on 4fwvko.dae4a70 Ocean Restart file copy failed on 4fwvko.dae4a80 Ocean Restart file copy failed on 4fwvko.dae4bj0 Ocean Restart file copy failed on 4fwvko.dae4bk0 Ocean Restart file copy failed on 4fwvko.dae4bl0 Ocean Restart file copy failed on 4fwvko.dae4bm0 Ocean Restart file copy failed on 4fwvko.dae4bn0 Ocean Restart file copy failed on 4fwvko.dae4bo0 Ocean Restart file copy failed on 4fwvko.dae4bp0 Ocean Restart file copy failed on 4fwvko.dae4bq0 Ocean Restart file copy failed on 4fwvko.dae4br0 Ocean Restart file copy failed on 4fwvko.dae4bs0 Ocean Restart file copy failed on 4fwvko.dae4bt0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae4ck0 Ocean Restart file copy failed on 4fwvko.dae4cl0 Ocean Restart file copy failed on 4fwvko.dae4cm0 Ocean Restart file copy failed on 4fwvko.dae4cn0 Ocean Restart file copy failed on 4fwvko.dae4co0 Ocean Restart file copy failed on 4fwvko.dae4cp0 Ocean Restart file copy failed on 4fwvko.dae4cq0 Ocean Restart file copy failed on 4fwvko.dae4cr0 Ocean Restart file copy failed on 4fwvko.dae4cs0 Ocean Restart file copy failed on 4fwvko.dae4ct0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 4fwvko.dae52n0 Ocean Restart file copy failed on 4fwvko.dae52o0 Ocean Restart file copy failed on 4fwvko.dae52p0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae5320 Ocean Restart file copy failed on 4fwvko.dae5330 Ocean Restart file copy failed on 4fwvko.dae5340 Ocean Restart file copy failed on 4fwvko.dae5350 Ocean Restart file copy failed on 4fwvko.dae5360 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae5450 Ocean Restart file copy failed on 4fwvko.dae5460 Ocean Restart file copy failed on 4fwvko.dae5470 Ocean Restart file copy failed on 4fwvko.dae5480 Ocean Restart file copy failed on 4fwvko.dae5490 Ocean Restart file copy failed on 4fwvko.dae54a0 Ocean Restart file copy failed on 4fwvko.dae54b0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae54o0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae54p0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae58k0 Ocean Restart file copy failed on 4fwvko.dae58l0 Ocean Restart file copy failed on 4fwvko.dae58m0 Ocean Restart file copy failed on 4fwvko.dae58n0 Ocean Restart file copy failed on 4fwvko.dae58o0 Ocean Restart file copy failed on 4fwvko.dae58p0 Ocean Restart file copy failed on 4fwvko.dae58q0 Ocean Restart file copy failed on 4fwvko.dae58r0 Ocean Restart file copy failed on 4fwvko.dae58s0 Ocean Restart file copy failed on 4fwvko.dae59f0 Ocean Restart file copy failed on 4fwvko.dae59g0 Ocean Restart file copy failed on 4fwvko.dae59h0 Ocean Restart file copy failed on 4fwvko.dae59i0 Ocean Restart file copy failed on 4fwvko.dae59j0 Ocean Restart file copy failed on 4fwvko.dae59k0 Ocean Restart file copy failed on 4fwvko.dae59l0 Ocean Restart file copy failed on 4fwvko.dae59m0 Ocean Restart file copy failed on 4fwvko.dae59n0 Ocean Restart file copy failed on 4fwvko.dae59o0 Ocean Restart file copy failed on 4fwvko.dae59p0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae59r0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae59s0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4fwvka.dae59t0 Ocean Restart file copy failed on 4fwvko.dae5bi0 Ocean Restart file copy failed on 4fwvko.dae5bj0 Ocean Restart file copy failed on 4fwvko.dae5bk0 Ocean Restart file copy failed on 4fwvko.dae5bl0 Ocean Restart file copy failed on 4fwvko.dae5bm0 Ocean Restart file copy failed on 4fwvko.dae5bn0 Ocean Restart file copy failed on 4fwvko.dae5bo0 Ocean Restart file copy failed on 4fwvko.dae5bp0 Ocean Restart file copy failed on 4fwvko.dae5bq0 Ocean Restart file copy failed on 4fwvko.dae5br0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae5cn0 Ocean Restart file copy failed on 4fwvko.dae5co0 Ocean Restart file copy failed on 4fwvko.dae5cp0 Ocean Restart file copy failed on 4fwvko.dae5cq0 Ocean Restart file copy failed on 4fwvko.dae5cr0 Ocean Restart file copy failed on 4fwvko.dae5cs0 Ocean Restart file copy failed on 4fwvko.dae5ct0 Ocean Restart file copy failed on 4fwvko.dae5cu0 Ocean Restart file copy failed on 4fwvko.dae6110 Ocean Restart file copy failed on 4fwvko.dae6120 Ocean Restart file copy failed on 4fwvko.dae6130 Ocean Restart file copy failed on 4fwvko.dae61e0 Ocean Restart file copy failed on 4fwvko.dae61f0 Ocean Restart file copy failed on 4fwvko.dae61g0 Ocean Restart file copy failed on 4fwvko.dae61h0 Ocean Restart file copy failed on 4fwvko.dae61i0 Ocean Restart file copy failed on 4fwvko.dae61j0 Ocean Restart file copy failed on 4fwvko.dae61k0 Ocean Restart file copy failed on 4fwvko.dae61l0 Ocean Restart file copy failed on 4fwvko.dae61m0 Ocean Restart file copy failed on 4fwvko.dae61n0 Ocean Restart file copy failed on 4fwvko.dae61o0 Ocean Restart file copy failed on 4fwvko.dae61p0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae6350 Ocean Restart file copy failed on 4fwvko.dae6360 Ocean Restart file copy failed on 4fwvko.dae6370 Ocean Restart file copy failed on 4fwvko.dae6380 Ocean Restart file copy failed on 4fwvko.dae6390 Ocean Restart file copy failed on 4fwvko.dae63a0 Ocean Restart file copy failed on 4fwvko.dae63b0 Ocean Restart file copy failed on 4fwvko.dae63c0 Ocean Restart file copy failed on 4fwvko.dae63d0 Ocean Restart file copy failed on 4fwvko.dae63e0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae63k0 Ocean Restart file copy failed on 4fwvko.dae63l0 Ocean Restart file copy failed on 4fwvko.dae63m0 Ocean Restart file copy failed on 4fwvko.dae63n0 Ocean Restart file copy failed on 4fwvko.dae63o0 Ocean Restart file copy failed on 4fwvko.dae63p0 Ocean Restart file copy failed on 4fwvko.dae63q0 Ocean Restart file copy failed on 4fwvko.dae63r0 Ocean Restart file copy failed on 4fwvko.dae63s0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae6450 Ocean Restart file copy failed on 4fwvko.dae6460 Ocean Restart file copy failed on 4fwvko.dae6470 Ocean Restart file copy failed on 4fwvko.dae6480 Ocean Restart file copy failed on 4fwvko.dae6490 CPDN Monitor - Quit request from BOINC... 11:07:32 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:20 (8880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:42 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 4fwvko.dae79o0 Ocean Restart file copy failed on 4fwvko.dae79p0 Ocean Restart file copy failed on 4fwvko.dae79q0 Ocean Restart file copy failed on 4fwvko.dae79r0 Ocean Restart file copy failed on 4fwvko.dae79s0 Ocean Restart file copy failed on 4fwvko.dae79t0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae7a70 Ocean Restart file copy failed on 4fwvko.dae7a80 Ocean Restart file copy failed on 4fwvko.dae7a90 Ocean Restart file copy failed on 4fwvko.dae7aa0 Ocean Restart file copy failed on 4fwvko.dae7ab0 Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4740, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running09:40:41 (2056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 4fwvko.dae8170 Ocean Restart file copy failed on 4fwvko.dae8180 Ocean Restart file copy failed on 4fwvko.dae8190 Ocean Restart file copy failed on 4fwvko.dae81a0 Ocean Restart file copy failed on 4fwvko.dae81b0 Ocean Restart file copy failed on 4fwvko.dae81c0 Ocean Restart file copy failed on 4fwvko.dae81d0 Ocean Restart file copy failed on 4fwvko.dae81e0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 4fwvko.dae81o0 Ocean Restart file copy failed on 4fwvko.dae81p0 Ocean Restart file copy failed on 4fwvko.dae81q0 Ocean Restart file copy failed on 4fwvko.dae81r0 Ocean Restart file copy failed on 4fwvko.dae81s0 Ocean Restart file copy failed on 4fwvko.dae81t0 Ocean Restart file copy failed on 4fwvko.dae81u0 Ocean Restart file copy failed on 4fwvko.dae8210 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:41:05 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:02:05 (9696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:22:19 (2120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:22:20 (2120): No heartbeat from core client for 30 sec - exiting 13:22:21 (2120): No heartbeat from core client for 30 sec - exiting 13:22:22 (2120): No heartbeat from core client for 30 sec - exiting 13:22:23 (2120): No heartbeat from core client for 30 sec - exiting 13:22:24 (2120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9448, iMonCtr=1 Model crash detected, will try to restart... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2013 23:47:49	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	285,120	830,312	2.9121
07 Jul 2013 07:20:45	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	259,200	750,835	2.8967
06 Jul 2013 05:29:37	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	233,280	673,589	2.8875
04 Jul 2013 14:26:46	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	207,360	595,699	2.8728
02 Jul 2013 10:38:16	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	181,440	517,384	2.8515
28 Jun 2013 05:22:04	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	155,520	432,561	2.7814
26 Jun 2013 07:03:49	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	129,600	354,917	2.7386
25 Jun 2013 03:43:56	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	103,680	289,756	2.7947
23 Jun 2013 19:36:15	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	77,760	222,639	2.8632
22 Jun 2013 17:31:31	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	51,840	151,528	2.9230
21 Jun 2013 15:51:36	1215740	15802561	hadcm3n_4fwv_1940_40_008302790_4	25,920	74,728	2.8830