Task 14840893

Name	hadcm3n_o6tk_2060_40_008026001_0
Workunit	8181115
Created	25 Jun 2012, 2:40:53 UTC
Sent	25 Jun 2012, 2:41:06 UTC
Report deadline	24 Sep 2012, 10:08:17 UTC
Received	6 Jul 2012, 7:03:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1099619
Run time	3 days 22 hours 40 min 11 sec
CPU time	3 days 12 hours 11 min 56 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	3.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:21:59 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:36:23 (11160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:24 (11160): No heartbeat from core client for 30 sec - exiting 15:36:25 (11160): No heartbeat from core client for 30 sec - exiting 15:36:26 (11160): No heartbeat from core client for 30 sec - exiting 15:36:27 (11160): No heartbeat from core client for 30 sec - exiting 15:36:28 (11160): No heartbeat from core client for 30 sec - exiting 15:36:29 (11160): No heartbeat from core client for 30 sec - exiting 15:36:30 (11160): No heartbeat from core client for 30 sec - exiting 15:36:31 (11160): No heartbeat from core client for 30 sec - exiting 15:36:32 (11160): No heartbeat from core client for 30 sec - exiting 15:36:33 (11160): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:25:59 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:26:00 (6384): No heartbeat from core client for 30 sec - exiting 17:26:01 (6384): No heartbeat from core client for 30 sec - exiting 17:26:02 (6384): No heartbeat from core client for 30 sec - exiting 17:26:03 (6384): No heartbeat from core client for 30 sec - exiting 17:26:04 (6384): No heartbeat from core client for 30 sec - exiting 17:26:05 (6384): No heartbeat from core client for 30 sec - exiting 17:26:06 (6384): No heartbeat from core client for 30 sec - exiting 17:26:07 (6384): No heartbeat from core client for 30 sec - exiting 17:26:08 (6384): No heartbeat from core client for 30 sec - exiting 17:26:09 (6384): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:37:26 (7420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:37:27 (7420): No heartbeat from core client for 30 sec - exiting 17:37:28 (7420): No heartbeat from core client for 30 sec - exiting 17:37:29 (7420): No heartbeat from core client for 30 sec - exiting 17:37:30 (7420): No heartbeat from core client for 30 sec - exiting 17:37:31 (7420): No heartbeat from core client for 30 sec - exiting 17:37:32 (7420): No heartbeat from core client for 30 sec - exiting 17:37:33 (7420): No heartbeat from core client for 30 sec - exiting 17:37:34 (7420): No heartbeat from core client for 30 sec - exiting 17:37:35 (7420): No heartbeat from core client for 30 sec - exiting 17:37:36 (7420): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:57:58 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:57:59 (5044): No heartbeat from core client for 30 sec - exiting 17:58:00 (5044): No heartbeat from core client for 30 sec - exiting 17:58:01 (5044): No heartbeat from core client for 30 sec - exiting 17:58:02 (5044): No heartbeat from core client for 30 sec - exiting 17:58:03 (5044): No heartbeat from core client for 30 sec - exiting 17:58:04 (5044): No heartbeat from core client for 30 sec - exiting 17:58:05 (5044): No heartbeat from core client for 30 sec - exiting 17:58:06 (5044): No heartbeat from core client for 30 sec - exiting 17:58:07 (5044): No heartbeat from core client for 30 sec - exiting 17:58:08 (5044): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:13:00 (9084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:01 (9084): No heartbeat from core client for 30 sec - exiting 18:13:02 (9084): No heartbeat from core client for 30 sec - exiting 18:13:03 (9084): No heartbeat from core client for 30 sec - exiting 18:13:04 (9084): No heartbeat from core client for 30 sec - exiting 18:13:05 (9084): No heartbeat from core client for 30 sec - exiting 18:13:06 (9084): No heartbeat from core client for 30 sec - exiting 18:13:07 (9084): No heartbeat from core client for 30 sec - exiting 18:13:08 (9084): No heartbeat from core client for 30 sec - exiting 18:13:09 (9084): No heartbeat from core client for 30 sec - exiting 18:13:10 (9084): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:33:02 (8756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:03 (8756): No heartbeat from core client for 30 sec - exiting 18:33:04 (8756): No heartbeat from core client for 30 sec - exiting 18:33:05 (8756): No heartbeat from core client for 30 sec - exiting 18:33:06 (8756): No heartbeat from core client for 30 sec - exiting 18:33:07 (8756): No heartbeat from core client for 30 sec - exiting 18:33:08 (8756): No heartbeat from core client for 30 sec - exiting 18:33:09 (8756): No heartbeat from core client for 30 sec - exiting 18:33:10 (8756): No heartbeat from core client for 30 sec - exiting 18:33:11 (8756): No heartbeat from core client for 30 sec - exiting 18:33:12 (8756): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:23:02 (9136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:03 (9136): No heartbeat from core client for 30 sec - exiting 21:23:04 (9136): No heartbeat from core client for 30 sec - exiting 21:23:05 (9136): No heartbeat from core client for 30 sec - exiting 21:23:06 (9136): No heartbeat from core client for 30 sec - exiting 21:23:07 (9136): No heartbeat from core client for 30 sec - exiting 21:23:08 (9136): No heartbeat from core client for 30 sec - exiting 21:23:09 (9136): No heartbeat from core client for 30 sec - exiting 21:23:10 (9136): No heartbeat from core client for 30 sec - exiting 21:23:11 (9136): No heartbeat from core client for 30 sec - exiting 21:23:12 (9136): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:20:05 (8984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:13:02 (8340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:03 (8340): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:14:37 (8948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:35:40 (9188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:35:41 (9188): No heartbeat from core client for 30 sec - exiting 00:35:42 (9188): No heartbeat from core client for 30 sec - exiting 00:35:43 (9188): No heartbeat from core client for 30 sec - exiting 00:35:44 (9188): No heartbeat from core client for 30 sec - exiting 00:35:45 (9188): No heartbeat from core client for 30 sec - exiting 00:35:46 (9188): No heartbeat from core client for 30 sec - exiting 00:35:47 (9188): No heartbeat from core client for 30 sec - exiting 00:35:48 (9188): No heartbeat from core client for 30 sec - exiting 00:35:49 (9188): No heartbeat from core client for 30 sec - exiting 00:35:50 (9188): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 01:11:57 (10656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:11:58 (10656): No heartbeat from core client for 30 sec - exiting 01:11:59 (10656): No heartbeat from core client for 30 sec - exiting 01:12:00 (10656): No heartbeat from core client for 30 sec - exiting 01:12:01 (10656): No heartbeat from core client for 30 sec - exiting 01:12:02 (10656): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:12:49 (11856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:50 (11856): No heartbeat from core client for 30 sec - exiting 21:12:51 (11856): No heartbeat from core client for 30 sec - exiting 21:12:52 (11856): No heartbeat from core client for 30 sec - exiting 21:12:53 (11856): No heartbeat from core client for 30 sec - exiting 21:12:54 (11856): No heartbeat from core client for 30 sec - exiting 21:12:55 (11856): No heartbeat from core client for 30 sec - exiting 21:12:56 (11856): No heartbeat from core client for 30 sec - exiting 21:12:57 (11856): No heartbeat from core client for 30 sec - exiting 21:12:58 (11856): No heartbeat from core client for 30 sec - exiting 21:12:59 (11856): No heartbeat from core client for 30 sec - exiting 21:29:51 (10696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:52 (10696): No heartbeat from core client for 30 sec - exiting 21:29:53 (10696): No heartbeat from core client for 30 sec - exiting 21:29:54 (10696): No heartbeat from core client for 30 sec - exiting 21:29:55 (10696): No heartbeat from core client for 30 sec - exiting 21:29:56 (10696): No heartbeat from core client for 30 sec - exiting 21:29:57 (10696): No heartbeat from core client for 30 sec - exiting 21:29:58 (10696): No heartbeat from core client for 30 sec - exiting 21:29:59 (10696): No heartbeat from core client for 30 sec - exiting 21:30:00 (10696): No heartbeat from core client for 30 sec - exiting 21:30:01 (10696): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:41:15 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:41:16 (5560): No heartbeat from core client for 30 sec - exiting 21:41:17 (5560): No heartbeat from core client for 30 sec - exiting 21:41:18 (5560): No heartbeat from core client for 30 sec - exiting 21:41:19 (5560): No heartbeat from core client for 30 sec - exiting 21:41:20 (5560): No heartbeat from core client for 30 sec - exiting 21:41:21 (5560): No heartbeat from core client for 30 sec - exiting 21:41:22 (5560): No heartbeat from core client for 30 sec - exiting 21:41:23 (5560): No heartbeat from core client for 30 sec - exiting 21:41:24 (5560): No heartbeat from core client for 30 sec - exiting 21:41:25 (5560): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7652, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7652, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7652, iMonCtr=1 Model crash detected, will try to restart... 22:01:52 (7652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:06 (8448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jul 2012 15:44:41	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	181,440	300,265	1.6549
04 Jul 2012 14:17:54	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	155,520	258,622	1.6630
03 Jul 2012 12:18:42	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	129,600	216,220	1.6684
02 Jul 2012 21:05:46	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	103,680	175,752	1.6951
26 Jun 2012 18:37:22	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	77,760	134,936	1.7353
26 Jun 2012 07:44:36	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	51,840	96,023	1.8523
25 Jun 2012 16:49:33	1099619	14840893	hadcm3n_o6tk_2060_40_008026001_0	25,920	47,753	1.8423