Task 15815063

Name	hadcm3n_o33s_1940_40_008383356_0
Workunit	8534215
Created	1 Jun 2013, 7:59:17 UTC
Sent	1 Jun 2013, 16:52:20 UTC
Report deadline	1 Sep 2013, 0:19:31 UTC
Received	22 Oct 2013, 14:33:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1204099
Run time	28 days 3 hours 32 min 36 sec
CPU time	27 days 6 hours 33 min 1 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:20:37 (12252): No heartbeat from core client for 30 sec - exiting 17:20:38 (12252): No heartbeat from core client for 30 sec - exiting 17:20:39 (12252): No heartbeat from core client for 30 sec - exiting 17:20:40 (12252): No heartbeat from core client for 30 sec - exiting 17:20:41 (12252): No heartbeat from core client for 30 sec - exiting 17:20:42 (12252): No heartbeat from core client for 30 sec - exiting 17:20:43 (12252): No heartbeat from core client for 30 sec - exiting 17:20:44 (12252): No heartbeat from core client for 30 sec - exiting 17:20:45 (12252): No heartbeat from core client for 30 sec - exiting 17:20:46 (12252): No heartbeat from core client for 30 sec - exiting 17:20:47 (12252): No heartbeat from core client for 30 sec - exiting 17:20:48 (12252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2144, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1076, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5436, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5436, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:45:34 (1956): No heartbeat from core client for 30 sec - exiting 08:45:35 (1956): No heartbeat from core client for 30 sec - exiting 08:45:36 (1956): No heartbeat from core client for 30 sec - exiting 08:45:37 (1956): No heartbeat from core client for 30 sec - exiting 08:45:38 (1956): No heartbeat from core client for 30 sec - exiting 08:45:39 (1956): No heartbeat from core client for 30 sec - exiting 08:45:40 (1956): No heartbeat from core client for 30 sec - exiting 08:45:41 (1956): No heartbeat from core client for 30 sec - exiting 08:45:42 (1956): No heartbeat from core client for 30 sec - exiting 08:45:44 (1956): No heartbeat from core client for 30 sec - exiting 08:45:45 (1956): No heartbeat from core client for 30 sec - exiting 08:45:46 (1956): No heartbeat from core client for 30 sec - exiting 08:45:47 (1956): No heartbeat from core client for 30 sec - exiting 08:45:48 (1956): No heartbeat from core client for 30 sec - exiting 08:45:49 (1956): No heartbeat from core client for 30 sec - exiting 08:45:50 (1956): No heartbeat from core client for 30 sec - exiting 08:45:51 (1956): No heartbeat from core client for 30 sec - exiting 08:45:52 (1956): No heartbeat from core client for 30 sec - exiting 08:45:53 (1956): No heartbeat from core client for 30 sec - exiting 08:45:54 (1956): No heartbeat from core client for 30 sec - exiting 08:45:56 (1956): No heartbeat from core client for 30 sec - exiting 08:45:57 (1956): No heartbeat from core client for 30 sec - exiting 08:45:58 (1956): No heartbeat from core client for 30 sec - exiting 08:45:59 (1956): No heartbeat from core client for 30 sec - exiting 08:46:00 (1956): No heartbeat from core client for 30 sec - exiting 08:46:01 (1956): No heartbeat from core client for 30 sec - exiting 08:46:02 (1956): No heartbeat from core client for 30 sec - exiting 08:46:03 (1956): No heartbeat from core client for 30 sec - exiting 08:46:04 (1956): No heartbeat from core client for 30 sec - exiting 08:46:05 (1956): No heartbeat from core client for 30 sec - exiting 08:46:06 (1956): No heartbeat from core client for 30 sec - exiting 08:46:08 (1956): No heartbeat from core client for 30 sec - exiting 08:46:09 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:46:34 (4592): No heartbeat from core client for 30 sec - exiting 08:46:36 (4592): No heartbeat from core client for 30 sec - exiting 08:46:37 (4592): No heartbeat from core client for 30 sec - exiting 08:46:38 (4592): No heartbeat from core client for 30 sec - exiting 08:46:39 (4592): No heartbeat from core client for 30 sec - exiting 08:46:40 (4592): No heartbeat from core client for 30 sec - exiting 08:46:41 (4592): No heartbeat from core client for 30 sec - exiting 08:46:42 (4592): No heartbeat from core client for 30 sec - exiting 08:46:43 (4592): No heartbeat from core client for 30 sec - exiting 08:46:44 (4592): No heartbeat from core client for 30 sec - exiting 08:46:45 (4592): No heartbeat from core client for 30 sec - exiting 08:46:46 (4592): No heartbeat from core client for 30 sec - exiting 08:46:48 (4592): No heartbeat from core client for 30 sec - exiting 08:46:49 (4592): No heartbeat from core client for 30 sec - exiting 08:46:50 (4592): No heartbeat from core client for 30 sec - exiting 08:46:51 (4592): No heartbeat from core client for 30 sec - exiting 08:46:52 (4592): No heartbeat from core client for 30 sec - exiting 08:46:53 (4592): No heartbeat from core client for 30 sec - exiting 08:46:54 (4592): No heartbeat from core client for 30 sec - exiting 08:46:55 (4592): No heartbeat from core client for 30 sec - exiting 08:46:56 (4592): No heartbeat from core client for 30 sec - exiting 08:46:57 (4592): No heartbeat from core client for 30 sec - exiting 08:46:58 (4592): No heartbeat from core client for 30 sec - exiting 08:46:59 (4592): No heartbeat from core client for 30 sec - exiting 08:47:01 (4592): No heartbeat from core client for 30 sec - exiting 08:47:02 (4592): No heartbeat from core client for 30 sec - exiting 08:47:03 (4592): No heartbeat from core client for 30 sec - exiting 08:47:04 (4592): No heartbeat from core client for 30 sec - exiting 08:47:05 (4592): No heartbeat from core client for 30 sec - exiting 08:47:06 (4592): No heartbeat from core client for 30 sec - exiting 08:47:07 (4592): No heartbeat from core client for 30 sec - exiting 08:47:08 (4592): No heartbeat from core client for 30 sec - exiting 08:47:09 (4592): No heartbeat from core client for 30 sec - exiting 08:47:10 (4592): No heartbeat from core client for 30 sec - exiting 08:47:11 (4592): No heartbeat from core client for 30 sec - exiting 08:47:13 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:57:43 (1360): No heartbeat from core client for 30 sec - exiting 08:57:44 (1360): No heartbeat from core client for 30 sec - exiting 08:57:45 (1360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:35:05 (4240): No heartbeat from core client for 30 sec - exiting 10:35:06 (4240): No heartbeat from core client for 30 sec - exiting 10:35:07 (4240): No heartbeat from core client for 30 sec - exiting 10:35:08 (4240): No heartbeat from core client for 30 sec - exiting 10:35:09 (4240): No heartbeat from core client for 30 sec - exiting 10:35:10 (4240): No heartbeat from core client for 30 sec - exiting 10:35:12 (4240): No heartbeat from core client for 30 sec - exiting 10:35:13 (4240): No heartbeat from core client for 30 sec - exiting 10:35:14 (4240): No heartbeat from core client for 30 sec - exiting 10:35:15 (4240): No heartbeat from core client for 30 sec - exiting 10:35:16 (4240): No heartbeat from core client for 30 sec - exiting 10:35:17 (4240): No heartbeat from core client for 30 sec - exiting 10:35:18 (4240): No heartbeat from core client for 30 sec - exiting 10:35:19 (4240): No heartbeat from core client for 30 sec - exiting 10:35:20 (4240): No heartbeat from core client for 30 sec - exiting 10:35:21 (4240): No heartbeat from core client for 30 sec - exiting 10:35:22 (4240): No heartbeat from core client for 30 sec - exiting 10:35:24 (4240): No heartbeat from core client for 30 sec - exiting 10:35:12 (4240): No heartbeat from core client for 30 sec - exiting 10:35:13 (4240): No heartbeat from core client for 30 sec - exiting 10:35:14 (4240): No heartbeat from core client for 30 sec - exiting 10:35:15 (4240): No heartbeat from core client for 30 sec - exiting 10:35:16 (4240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:22:04 (4320): No heartbeat from core client for 30 sec - exiting 09:22:05 (4320): No heartbeat from core client for 30 sec - exiting 09:22:06 (4320): No heartbeat from core client for 30 sec - exiting 09:22:07 (4320): No heartbeat from core client for 30 sec - exiting 09:22:08 (4320): No heartbeat from core client for 30 sec - exiting 09:22:09 (4320): No heartbeat from core client for 30 sec - exiting 09:22:10 (4320): No heartbeat from core client for 30 sec - exiting 09:22:11 (4320): No heartbeat from core client for 30 sec - exiting 09:22:12 (4320): No heartbeat from core client for 30 sec - exiting 09:22:14 (4320): No heartbeat from core client for 30 sec - exiting 09:22:15 (4320): No heartbeat from core client for 30 sec - exiting 09:22:16 (4320): No heartbeat from core client for 30 sec - exiting 09:22:17 (4320): No heartbeat from core client for 30 sec - exiting 09:22:18 (4320): No heartbeat from core client for 30 sec - exiting 09:22:19 (4320): No heartbeat from core client for 30 sec - exiting 09:22:20 (4320): No heartbeat from core client for 30 sec - exiting 09:22:21 (4320): No heartbeat from core client for 30 sec - exiting 09:22:22 (4320): No heartbeat from core client for 30 sec - exiting 09:22:23 (4320): No heartbeat from core client for 30 sec - exiting 09:22:25 (4320): No heartbeat from core client for 30 sec - exiting 09:22:26 (4320): No heartbeat from core client for 30 sec - exiting 09:22:27 (4320): No heartbeat from core client for 30 sec - exiting 09:22:28 (4320): No heartbeat from core client for 30 sec - exiting 09:22:29 (4320): No heartbeat from core client for 30 sec - exiting 09:22:30 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:31 (4320): No heartbeat from core client for 30 sec - exiting 09:22:32 (4320): No heartbeat from core client for 30 sec - exiting 09:22:33 (4320): No heartbeat from core client for 30 sec - exiting 09:22:34 (4320): No heartbeat from core client for 30 sec - exiting 09:22:35 (4320): No heartbeat from core client for 30 sec - exiting 09:22:37 (4320): No heartbeat from core client for 30 sec - exiting 09:22:38 (4320): No heartbeat from core client for 30 sec - exiting 09:22:39 (4320): No heartbeat from core client for 30 sec - exiting 09:22:40 (4320): No heartbeat from core client for 30 sec - exiting 09:22:41 (4320): No heartbeat from core client for 30 sec - exiting 09:23:03 (2204): Can't acquire lockfile (32) - waiting 35s 09:23:38 (2204): Can't set up shared mem: -1. Will run in standalone mode. 09:23:38 (4276): Can't set up shared mem: -1. Will run in standalone mode. Signal 22 received, exiting... 09:23:47 (4208): Can't acquire lockfile (32) - waiting 35s 11:07:06 (1112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4596, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 16:28:39 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5048, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5048, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 17:08:38 (3380): No heartbeat from core client for 30 sec - exiting 17:08:39 (3380): No heartbeat from core client for 30 sec - exiting 17:08:40 (3380): No heartbeat from core client for 30 sec - exiting 17:08:41 (3380): No heartbeat from core client for 30 sec - exiting 17:08:42 (3380): No heartbeat from core client for 30 sec - exiting 17:08:43 (3380): No heartbeat from core client for 30 sec - exiting 17:08:44 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:45 (3380): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jul 2013 19:29:15	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	907,200	1,792,494	1.9759
10 Jul 2013 04:34:35	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	881,280	1,740,363	1.9748
09 Jul 2013 14:13:50	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	855,360	1,689,554	1.9753
08 Jul 2013 23:37:32	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	829,440	1,638,013	1.9748
08 Jul 2013 08:32:15	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	803,520	1,584,936	1.9725
07 Jul 2013 17:31:38	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	777,600	1,531,877	1.9700
07 Jul 2013 01:56:46	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	751,680	1,479,158	1.9678
06 Jul 2013 10:24:28	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	725,760	1,426,441	1.9654
06 Jul 2013 05:26:02	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	699,840	1,375,983	1.9661
06 Jul 2013 04:14:11	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	673,920	1,325,158	1.9663
03 Jul 2013 17:24:22	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	648,000	1,274,262	1.9665
02 Jul 2013 16:57:14	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	622,080	1,220,960	1.9627
02 Jul 2013 11:52:07	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	596,160	1,168,942	1.9608
02 Jul 2013 10:57:02	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	570,240	1,117,228	1.9592
02 Jul 2013 10:31:22	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	544,320	1,067,688	1.9615
02 Jul 2013 10:12:04	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	518,400	1,019,055	1.9658
02 Jul 2013 09:55:09	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	492,480	970,418	1.9705
28 Jun 2013 03:21:09	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	466,560	921,402	1.9749
27 Jun 2013 03:46:49	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	440,640	869,471	1.9732
26 Jun 2013 02:41:52	1204099	15815063	hadcm3n_o33s_1940_40_008383356_0	414,720	815,095	1.9654