Task 15819177

Name	hadcm3n_o5nv_1980_40_008384298_0
Workunit	8535157
Created	2 Jun 2013, 23:10:50 UTC
Sent	13 Jun 2013, 3:24:11 UTC
Report deadline	12 Sep 2013, 10:51:22 UTC
Received	16 Sep 2013, 2:39:43 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID	1266922
Run time	32 days 4 hours 19 min 35 sec
CPU time	27 days 18 hours 41 min 30 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	1.05 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4624, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4740, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4656, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2772, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4924, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4268, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4892, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3056, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6196, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3300, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4104, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4996, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4844, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1304, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=496, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on o5nvko.daj97f0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... 17:42:23 (4160): No heartbeat from core client for 30 sec - exiting 17:42:24 (4160): No heartbeat from core client for 30 sec - exiting 17:42:25 (4160): No heartbeat from core client for 30 sec - exiting 17:42:27 (4160): No heartbeat from core client for 30 sec - exiting 17:42:28 (4160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:27:37 (4740): No heartbeat from core client for 30 sec - exiting 09:27:38 (4740): No heartbeat from core client for 30 sec - exiting 09:27:40 (4740): No heartbeat from core client for 30 sec - exiting 09:27:41 (4740): No heartbeat from core client for 30 sec - exiting 09:27:42 (4740): No heartbeat from core client for 30 sec - exiting 09:27:43 (4740): No heartbeat from core client for 30 sec - exiting 09:27:44 (4740): No heartbeat from core client for 30 sec - exiting 09:27:45 (4740): No heartbeat from core client for 30 sec - exiting 09:27:46 (4740): No heartbeat from core client for 30 sec - exiting 09:27:47 (4740): No heartbeat from core client for 30 sec - exiting 09:27:48 (4740): No heartbeat from core client for 30 sec - exiting 09:27:49 (4740): No heartbeat from core client for 30 sec - exiting 09:27:50 (4740): No heartbeat from core client for 30 sec - exiting 09:27:52 (4740): No heartbeat from core client for 30 sec - exiting 09:27:53 (4740): No heartbeat from core client for 30 sec - exiting 09:27:54 (4740): No heartbeat from core client for 30 sec - exiting 09:27:55 (4740): No heartbeat from core client for 30 sec - exiting 09:27:56 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:28:07 (5060): No heartbeat from core client for 30 sec - exiting 21:28:08 (5060): No heartbeat from core client for 30 sec - exiting 21:28:09 (5060): No heartbeat from core client for 30 sec - exiting 21:28:10 (5060): No heartbeat from core client for 30 sec - exiting 21:28:11 (5060): No heartbeat from core client for 30 sec - exiting 21:28:13 (5060): No heartbeat from core client for 30 sec - exiting 21:28:14 (5060): No heartbeat from core client for 30 sec - exiting 21:28:15 (5060): No heartbeat from core client for 30 sec - exiting 21:28:16 (5060): No heartbeat from core client for 30 sec - exiting 21:28:17 (5060): No heartbeat from core client for 30 sec - exiting 21:28:18 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3656, iMonCtr=1 Model crash detected, will try to restart... 10:42:43 (5040): No heartbeat from core client for 30 sec - exiting 10:42:45 (5040): No heartbeat from core client for 30 sec - exiting 10:42:46 (5040): No heartbeat from core client for 30 sec - exiting 10:42:47 (5040): No heartbeat from core client for 30 sec - exiting 10:42:48 (5040): No heartbeat from core client for 30 sec - exiting 10:42:49 (5040): No heartbeat from core client for 30 sec - exiting 10:42:50 (5040): No heartbeat from core client for 30 sec - exiting 10:42:51 (5040): No heartbeat from core client for 30 sec - exiting 10:42:52 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1704, iMonCtr=1 Model crash detected, will try to restart... 20:40:39 (1004): No heartbeat from core client for 30 sec - exiting 20:40:40 (1004): No heartbeat from core client for 30 sec - exiting 20:40:41 (1004): No heartbeat from core client for 30 sec - exiting 20:40:42 (1004): No heartbeat from core client for 30 sec - exiting 20:40:43 (1004): No heartbeat from core client for 30 sec - exiting 20:40:44 (1004): No heartbeat from core client for 30 sec - exiting 20:40:45 (1004): No heartbeat from core client for 30 sec - exiting 20:40:46 (1004): No heartbeat from core client for 30 sec - exiting 20:40:47 (1004): No heartbeat from core client for 30 sec - exiting 20:40:49 (1004): No heartbeat from core client for 30 sec - exiting 20:40:50 (1004): No heartbeat from core client for 30 sec - exiting 20:40:51 (1004): No heartbeat from core client for 30 sec - exiting 20:40:52 (1004): No heartbeat from core client for 30 sec - exiting 20:40:53 (1004): No heartbeat from core client for 30 sec - exiting 20:40:54 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... 18:21:01 (4284): No heartbeat from core client for 30 sec - exiting 18:21:02 (4284): No heartbeat from core client for 30 sec - exiting 18:21:03 (4284): No heartbeat from core client for 30 sec - exiting 18:21:04 (4284): No heartbeat from core client for 30 sec - exiting 18:21:06 (4284): No heartbeat from core client for 30 sec - exiting 18:21:07 (4284): No heartbeat from core client for 30 sec - exiting 18:21:08 (4284): No heartbeat from core client for 30 sec - exiting 18:21:09 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:02:42 (4456): No heartbeat from core client for 30 sec - exiting 19:02:43 (4456): No heartbeat from core client for 30 sec - exiting 19:02:44 (4456): No heartbeat from core client for 30 sec - exiting 19:02:45 (4456): No heartbeat from core client for 30 sec - exiting 19:02:46 (4456): No heartbeat from core client for 30 sec - exiting 19:02:47 (4456): No heartbeat from core client for 30 sec - exiting 19:02:48 (4456): No heartbeat from core client for 30 sec - exiting 19:02:49 (4456): No heartbeat from core client for 30 sec - exiting 19:02:50 (4456): No heartbeat from core client for 30 sec - exiting 19:02:51 (4456): No heartbeat from core client for 30 sec - exiting 19:02:52 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2168, iMonCtr=1 Model crash detected, will try to restart... 19:40:41 (3384): No heartbeat from core client for 30 sec - exiting 19:40:43 (3384): No heartbeat from core client for 30 sec - exiting 19:40:44 (3384): No heartbeat from core client for 30 sec - exiting 19:40:45 (3384): No heartbeat from core client for 30 sec - exiting 19:40:46 (3384): No heartbeat from core client for 30 sec - exiting 19:40:47 (3384): No heartbeat from core client for 30 sec - exiting 19:40:48 (3384): No heartbeat from core client for 30 sec - exiting 19:40:49 (3384): No heartbeat from core client for 30 sec - exiting 19:40:50 (3384): No heartbeat from core client for 30 sec - exiting 19:40:51 (3384): No heartbeat from core client for 30 sec - exiting 19:40:52 (3384): No heartbeat from core client for 30 sec - exiting 19:40:53 (3384): No heartbeat from core client for 30 sec - exiting 19:40:55 (3384): No heartbeat from core client for 30 sec - exiting 19:40:56 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:01:45 (5084): No heartbeat from core client for 30 sec - exiting 20:01:46 (5084): No heartbeat from core client for 30 sec - exiting 20:01:48 (5084): No heartbeat from core client for 30 sec - exiting 20:01:49 (5084): No heartbeat from core client for 30 sec - exiting 20:01:50 (5084): No heartbeat from core client for 30 sec - exiting 20:01:51 (5084): No heartbeat from core client for 30 sec - exiting 20:01:52 (5084): No heartbeat from core client for 30 sec - exiting 20:01:53 (5084): No heartbeat from core client for 30 sec - exiting 20:01:54 (5084): No heartbeat from core client for 30 sec - exiting 20:01:55 (5084): No heartbeat from core client for 30 sec - exiting 20:01:56 (5084): No heartbeat from core client for 30 sec - exiting 20:01:57 (5084): No heartbeat from core client for 30 sec - exiting 20:01:58 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:53:24 (4300): No heartbeat from core client for 30 sec - exiting 09:53:26 (4300): No heartbeat from core client for 30 sec - exiting 09:53:27 (4300): No heartbeat from core client for 30 sec - exiting 09:53:28 (4300): No heartbeat from core client for 30 sec - exiting 09:53:29 (4300): No heartbeat from core client for 30 sec - exiting 09:53:30 (4300): No heartbeat from core client for 30 sec - exiting 09:53:31 (4300): No heartbeat from core client for 30 sec - exiting 09:53:32 (4300): No heartbeat from core client for 30 sec - exiting 09:53:33 (4300): No heartbeat from core client for 30 sec - exiting 09:53:34 (4300): No heartbeat from core client for 30 sec - exiting 09:53:35 (4300): No heartbeat from core client for 30 sec - exiting 09:53:36 (4300): No heartbeat from core client for 30 sec - exiting 09:53:38 (4300): No heartbeat from core client for 30 sec - exiting 09:53:39 (4300): No heartbeat from core client for 30 sec - exiting 09:53:40 (4300): No heartbeat from core client for 30 sec - exiting 09:53:41 (4300): No heartbeat from core client for 30 sec - exiting 09:53:42 (4300): No heartbeat from core client for 30 sec - exiting 09:53:43 (4300): No heartbeat from core client for 30 sec - exiting 09:53:44 (4300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=384, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Abort request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Sep 2013 01:12:04	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	673,920	2,359,298	3.5009
12 Sep 2013 10:50:03	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	648,000	2,263,136	3.4925
07 Sep 2013 16:56:18	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	622,080	2,164,714	3.4798
01 Sep 2013 08:49:08	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	596,160	2,076,310	3.4828
25 Aug 2013 08:48:03	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	570,240	1,989,357	3.4886
22 Aug 2013 07:44:16	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	544,320	1,901,095	3.4926
18 Aug 2013 07:24:28	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	518,400	1,803,732	3.4794
15 Aug 2013 07:15:07	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	492,480	1,699,361	3.4506
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	466,560	1,598,242	3.4256
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	440,640	1,494,749	3.3922
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	414,720	1,398,913	3.3732
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	388,800	1,299,884	3.3433
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	362,880	1,202,269	3.3131
14 Aug 2013 16:27:32	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	336,960	1,111,013	3.2972
29 Jul 2013 12:55:30	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	311,040	1,023,431	3.2904
23 Jul 2013 20:18:31	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	285,120	929,722	3.2608
23 Jul 2013 17:12:48	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	259,200	839,620	3.2393
23 Jul 2013 17:12:48	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	233,280	762,075	3.2668
10 Jul 2013 09:51:09	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	207,360	693,730	3.3455
07 Jul 2013 06:15:35	1266922	15819177	hadcm3n_o5nv_1980_40_008384298_0	181,440	625,454	3.4472