Task 15464259

Name	hadcm3n_zg5d_1880_40_008253607_2
Workunit	8408731
Created	27 Nov 2012, 10:08:41 UTC
Sent	27 Nov 2012, 10:08:51 UTC
Report deadline	26 Feb 2013, 17:36:02 UTC
Received	25 Dec 2012, 14:05:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1253823
Run time	14 days 15 hours 11 min 54 sec
CPU time	13 days 22 hours 14 min 7 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 02:14:23 (2068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:24 (2068): No heartbeat from core client for 30 sec - exiting 02:16:59 (3436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:00 (3436): No heartbeat from core client for 30 sec - exiting 02:17:03 (3436): No heartbeat from core client for 30 sec - exiting 02:17:05 (3436): No heartbeat from core client for 30 sec - exiting 02:17:06 (3436): No heartbeat from core client for 30 sec - exiting 02:32:28 (7472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:30 (7472): No heartbeat from core client for 30 sec - exiting 02:32:32 (7472): No heartbeat from core client for 30 sec - exiting 02:32:35 (7472): No heartbeat from core client for 30 sec - exiting 02:32:37 (7472): No heartbeat from core client for 30 sec - exiting 02:32:40 (7472): No heartbeat from core client for 30 sec - exiting 02:47:32 (10004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:33 (10004): No heartbeat from core client for 30 sec - exiting 03:20:41 (9976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:43 (9976): No heartbeat from core client for 30 sec - exiting 03:20:46 (9976): No heartbeat from core client for 30 sec - exiting 03:20:49 (9976): No heartbeat from core client for 30 sec - exiting 03:20:51 (9976): No heartbeat from core client for 30 sec - exiting 03:20:54 (9976): No heartbeat from core client for 30 sec - exiting 03:55:02 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:04 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 03:30:50 (5280): No heartbeat from core client for 30 sec - exiting 03:30:51 (5280): No heartbeat from core client for 30 sec - exiting 03:30:52 (5280): No heartbeat from core client for 30 sec - exiting 03:30:53 (5280): No heartbeat from core client for 30 sec - exiting 03:30:54 (5280): No heartbeat from core client for 30 sec - exiting 03:30:55 (5280): No heartbeat from core client for 30 sec - exiting 03:30:56 (5280): No heartbeat from core client for 30 sec - exiting 03:30:57 (5280): No heartbeat from core client for 30 sec - exiting 03:30:58 (5280): No heartbeat from core client for 30 sec - exiting 03:31:00 (5280): No heartbeat from core client for 30 sec - exiting 03:31:01 (5280): No heartbeat from core client for 30 sec - exiting 03:31:02 (5280): No heartbeat from core client for 30 sec - exiting 03:31:03 (5280): No heartbeat from core client for 30 sec - exiting 03:31:04 (5280): No heartbeat from core client for 30 sec - exiting 03:31:05 (5280): No heartbeat from core client for 30 sec - exiting 03:31:06 (5280): No heartbeat from core client for 30 sec - exiting 03:31:07 (5280): No heartbeat from core client for 30 sec - exiting 03:31:08 (5280): No heartbeat from core client for 30 sec - exiting 03:31:09 (5280): No heartbeat from core client for 30 sec - exiting 03:31:10 (5280): No heartbeat from core client for 30 sec - exiting 03:31:11 (5280): No heartbeat from core client for 30 sec - exiting 03:31:12 (5280): No heartbeat from core client for 30 sec - exiting 03:31:13 (5280): No heartbeat from core client for 30 sec - exiting 03:31:14 (5280): No heartbeat from core client for 30 sec - exiting 03:31:15 (5280): No heartbeat from core client for 30 sec - exiting 03:31:16 (5280): No heartbeat from core client for 30 sec - exiting 03:31:17 (5280): No heartbeat from core client for 30 sec - exiting 03:31:18 (5280): No heartbeat from core client for 30 sec - exiting 03:31:19 (5280): No heartbeat from core client for 30 sec - exiting 03:31:20 (5280): No heartbeat from core client for 30 sec - exiting 03:31:21 (5280): No heartbeat from core client for 30 sec - exiting 03:31:22 (5280): No heartbeat from core client for 30 sec - exiting 03:31:23 (5280): No heartbeat from core client for 30 sec - exiting 03:31:24 (5280): No heartbeat from core client for 30 sec - exiting 03:31:25 (5280): No heartbeat from core client for 30 sec - exiting 03:31:26 (5280): No heartbeat from core client for 30 sec - exiting 03:31:27 (5280): No heartbeat from core client for 30 sec - exiting 03:31:28 (5280): No heartbeat from core client for 30 sec - exiting 03:31:29 (5280): No heartbeat from core client for 30 sec - exiting 03:31:30 (5280): No heartbeat from core client for 30 sec - exiting 03:31:31 (5280): No heartbeat from core client for 30 sec - exiting 03:31:32 (5280): No heartbeat from core client for 30 sec - exiting 03:31:33 (5280): No heartbeat from core client for 30 sec - exiting 03:31:34 (5280): No heartbeat from core client for 30 sec - exiting 03:31:35 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:27:36 (6296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:27:37 (6296): No heartbeat from core client for 30 sec - exiting 06:27:39 (6296): No heartbeat from core client for 30 sec - exiting 06:32:52 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:54 (3752): No heartbeat from core client for 30 sec - exiting 06:32:56 (3752): No heartbeat from core client for 30 sec - exiting 06:32:58 (3752): No heartbeat from core client for 30 sec - exiting 06:33:00 (3752): No heartbeat from core client for 30 sec - exiting 06:36:35 (3808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:14 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:16 (5516): No heartbeat from core client for 30 sec - exiting 07:33:10 (7376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:12 (7376): No heartbeat from core client for 30 sec - exiting 07:52:52 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:54 (3168): No heartbeat from core client for 30 sec - exiting 07:55:35 (2796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:54 (7772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:44 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:47 (3548): No heartbeat from core client for 30 sec - exiting 08:14:33 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:35 (7408): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:05:05 (7752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:32 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:35 (5868): No heartbeat from core client for 30 sec - exiting 06:45:37 (5868): No heartbeat from core client for 30 sec - exiting 06:45:39 (5868): No heartbeat from core client for 30 sec - exiting 06:45:42 (5868): No heartbeat from core client for 30 sec - exiting 06:45:44 (5868): No heartbeat from core client for 30 sec - exiting 06:48:43 (136): No heartbeat from core client for 30 sec - exiting 06:48:44 (136): No heartbeat from core client for 30 sec - exiting 06:48:46 (136): No heartbeat from core client for 30 sec - exiting 06:48:48 (136): No heartbeat from core client for 30 sec - exiting 06:48:50 (136): No heartbeat from core client for 30 sec - exiting 06:48:52 (136): No heartbeat from core client for 30 sec - exiting 06:48:54 (136): No heartbeat from core client for 30 sec - exiting 06:48:58 (136): No heartbeat from core client for 30 sec - exiting 06:49:00 (136): No heartbeat from core client for 30 sec - exiting 06:49:02 (136): No heartbeat from core client for 30 sec - exiting 06:49:06 (136): No heartbeat from core client for 30 sec - exiting 06:49:11 (136): No heartbeat from core client for 30 sec - exiting 06:49:13 (136): No heartbeat from core client for 30 sec - exiting 06:49:16 (136): No heartbeat from core client for 30 sec - exiting 06:49:19 (136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:15 (5724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:16 (5724): No heartbeat from core client for 30 sec - exiting 07:06:18 (5724): No heartbeat from core client for 30 sec - exiting 07:06:20 (5724): No heartbeat from core client for 30 sec - exiting 07:06:23 (5724): No heartbeat from core client for 30 sec - exiting 07:06:25 (5724): No heartbeat from core client for 30 sec - exiting 07:07:47 (5788): No heartbeat from core client for 30 sec - exiting 07:07:48 (5788): No heartbeat from core client for 30 sec - exiting 07:07:50 (5788): No heartbeat from core client for 30 sec - exiting 07:07:51 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:31:55 (10036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:58 (10036): No heartbeat from core client for 30 sec - exiting 08:02:21 (10012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:23 (10012): No heartbeat from core client for 30 sec - exiting 08:30:57 (10120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:00 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:02 (5400): No heartbeat from core client for 30 sec - exiting 08:35:04 (5400): No heartbeat from core client for 30 sec - exiting 08:40:52 (9412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:53 (9412): No heartbeat from core client for 30 sec - exiting 08:45:20 (8328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:43 (5336): No heartbeat from core client for 30 sec - exiting 08:47:44 (5336): No heartbeat from core client for 30 sec - exiting 08:47:47 (5336): No heartbeat from core client for 30 sec - exiting 08:47:49 (5336): No heartbeat from core client for 30 sec - exiting 08:47:51 (5336): No heartbeat from core client for 30 sec - exiting 08:47:53 (5336): No heartbeat from core client for 30 sec - exiting 08:47:55 (5336): No heartbeat from core client for 30 sec - exiting 08:47:57 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:23 (3416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:25 (3416): No heartbeat from core client for 30 sec - exiting 09:13:35 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:37 (4304): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:11:23 (4612): No heartbeat from core client for 30 sec - exiting 16:11:24 (4612): No heartbeat from core client for 30 sec - exiting 16:11:25 (4612): No heartbeat from core client for 30 sec - exiting 16:11:26 (4612): No heartbeat from core client for 30 sec - exiting 16:11:27 (4612): No heartbeat from core client for 30 sec - exiting 16:11:28 (4612): No heartbeat from core client for 30 sec - exiting 16:11:29 (4612): No heartbeat from core client for 30 sec - exiting 16:11:30 (4612): No heartbeat from core client for 30 sec - exiting 16:11:31 (4612): No heartbeat from core client for 30 sec - exiting 16:11:32 (4612): No heartbeat from core client for 30 sec - exiting 16:11:33 (4612): No heartbeat from core client for 30 sec - exiting 16:11:35 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:02:57 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:08 (6720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:01:39 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:51:46 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:48 (4132): No heartbeat from core client for 30 sec - exiting 04:10:56 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:58 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:59 (5732): No heartbeat from core client for 30 sec - exiting 04:30:56 (2952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:58 (2952): No heartbeat from core client for 30 sec - exiting 04:38:35 (7088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:36 (7088): No heartbeat from core client for 30 sec - exiting 04:38:39 (7088): No heartbeat from core client for 30 sec - exiting 04:41:10 (3620): No heartbeat from core client for 30 sec - exiting 04:41:11 (3620): No heartbeat from core client for 30 sec - exiting 04:41:13 (3620): No heartbeat from core client for 30 sec - exiting 04:41:14 (3620): No heartbeat from core client for 30 sec - exiting 04:41:16 (3620): No heartbeat from core client for 30 sec - exiting 04:41:18 (3620): No heartbeat from core client for 30 sec - exiting 04:41:19 (3620): No heartbeat from core client for 30 sec - exiting 04:41:20 (3620): No heartbeat from core client for 30 sec - exiting 04:41:21 (3620): No heartbeat from core client for 30 sec - exiting 04:41:25 (3620): No heartbeat from core client for 30 sec - exiting 04:41:26 (3620): No heartbeat from core client for 30 sec - exiting 04:41:27 (3620): No heartbeat from core client for 30 sec - exiting 04:41:29 (3620): No heartbeat from core client for 30 sec - exiting 04:41:31 (3620): No heartbeat from core client for 30 sec - exiting 04:41:32 (3620): No heartbeat from core client for 30 sec - exiting 04:41:34 (3620): No heartbeat from core client for 30 sec - exiting 04:41:35 (3620): No heartbeat from core client for 30 sec - exiting 04:41:36 (3620): No heartbeat from core client for 30 sec - exiting 04:41:37 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:41:38 (3620): No heartbeat from core client for 30 sec - exiting 04:48:19 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:22 (5944): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:31:46 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Dec 2012 04:01:28	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	492,480	1,164,448	2.3645
24 Dec 2012 05:57:10	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	466,560	1,100,637	2.3590
23 Dec 2012 11:03:34	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	440,640	1,036,292	2.3518
22 Dec 2012 15:15:44	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	414,720	972,372	2.3446
21 Dec 2012 20:17:52	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	388,800	909,843	2.3401
21 Dec 2012 02:08:38	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	362,880	847,436	2.3353
20 Dec 2012 08:41:49	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	336,960	785,986	2.3326
19 Dec 2012 14:42:59	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	311,040	725,327	2.3319
18 Dec 2012 20:22:32	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	285,120	663,734	2.3279
18 Dec 2012 02:25:35	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	259,200	601,078	2.3190
17 Dec 2012 08:04:33	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	233,280	538,674	2.3091
16 Dec 2012 14:55:45	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	207,360	478,697	2.3085
15 Dec 2012 19:35:04	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	181,440	418,343	2.3057
15 Dec 2012 02:22:18	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	155,520	358,355	2.3042
14 Dec 2012 09:05:23	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	129,600	296,810	2.2902
13 Dec 2012 18:41:00	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	103,680	237,166	2.2875
13 Dec 2012 18:41:00	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	77,760	178,575	2.2965
28 Nov 2012 22:35:05	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	51,840	119,676	2.3086
28 Nov 2012 04:20:46	1253823	15464259	hadcm3n_zg5d_1880_40_008253607_2	25,920	60,894	2.3493