Task 16192272

Name	hadcm3n_y8w3_1980_40_008391840_4
Workunit	8542699
Created	1 Jan 2014, 15:20:59 UTC
Sent	1 Jan 2014, 15:21:04 UTC
Report deadline	2 Apr 2014, 22:48:15 UTC
Received	14 Jan 2014, 20:42:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1235271
Run time	6 days 3 hours 35 min 1 sec
CPU time	5 days 2 hours 32 min 11 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	3.12 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2148, iMonCtr=1 Model crash Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4252, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4688, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4060, iMonCtr=1 Model crash detected, will try to restart... 13:32:12 (3160): No heartbeat from core client for 30 sec - exiting 13:32:13 (3160): No heartbeat from core client for 30 sec - exiting 13:32:14 (3160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:15 (3160): No heartbeat from core client for 30 sec - exiting 13:32:16 (3160): No heartbeat from core client for 30 sec - exiting 13:32:19 (3160): No heartbeat from core client for 30 sec - exiting 13:32:20 (3160): No heartbeat from core client for 30 sec - exiting 13:32:21 (3160): No heartbeat from core client for 30 sec - exiting 13:32:23 (3160): No heartbeat from core client for 30 sec - exiting 13:32:24 (3160): No heartbeat from core client for 30 sec - exiting 13:32:25 (3160): No heartbeat from core client for 30 sec - exiting 13:32:26 (3160): No heartbeat from core client for 30 sec - exiting 13:32:27 (3160): No heartbeat from core client for 30 sec - exiting 13:32:28 (3160): No heartbeat from core client for 30 sec - exiting 13:32:29 (3160): No heartbeat from core client for 30 sec - exiting 13:32:30 (3160): No heartbeat from core client for 30 sec - exiting 13:32:31 (3160): No heartbeat from core client for 30 sec - exiting 13:32:32 (3160): No heartbeat from core client for 30 sec - exiting 13:32:34 (3160): No heartbeat from core client for 30 sec - exiting 13:32:35 (3160): No heartbeat from core client for 30 sec - exiting 13:32:36 (3160): No heartbeat from core client for 30 sec - exiting 13:32:39 (3160): No heartbeat from core client for 30 sec - exiting 13:32:40 (3160): No heartbeat from core client for 30 sec - exiting 13:32:41 (3160): No heartbeat from core client for 30 sec - exiting 13:32:43 (3160): No heartbeat from core client for 30 sec - exiting 13:32:44 (3160): No heartbeat from core client for 30 sec - exiting 13:32:45 (3160): No heartbeat from core client for 30 sec - exiting 13:32:47 (3160): No heartbeat from core client for 30 sec - exiting 13:32:48 (3160): No heartbeat from core client for 30 sec - exiting 13:32:49 (3160): No heartbeat from core client for 30 sec - exiting 13:32:50 (3160): No heartbeat from core client for 30 sec - exiting 13:32:51 (3160): No heartbeat from core client for 30 sec - exiting 13:32:53 (3160): No heartbeat from core client for 30 sec - exiting 13:32:54 (3160): No heartbeat from core client for 30 sec - exiting 13:32:55 (3160): No heartbeat from core client for 30 sec - exiting 13:32:56 (3160): No heartbeat from core client for 30 sec - exiting 13:32:57 (3160): No heartbeat from core client for 30 sec - exiting 13:32:58 (3160): No heartbeat from core client for 30 sec - exiting 13:32:59 (3160): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3584, iMonCtr=1 Model crash detected, will try to restart... 13:56:16 (3296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:56:17 (3296): No heartbeat from core client for 30 sec - exiting 13:56:18 (3296): No heartbeat from core client for 30 sec - exiting 13:56:19 (3296): No heartbeat from core client for 30 sec - exiting 13:56:20 (3296): No heartbeat from core client for 30 sec - exiting 13:56:21 (3296): No heartbeat from core client for 30 sec - exiting 13:56:22 (3296): No heartbeat from core client for 30 sec - exiting 13:56:23 (3296): No heartbeat from core client for 30 sec - exiting 13:56:25 (3296): No heartbeat from core client for 30 sec - exiting 13:56:26 (3296): No heartbeat from core client for 30 sec - exiting 13:56:27 (3296): No heartbeat from core client for 30 sec - exiting 13:56:28 (3296): No heartbeat from core client for 30 sec - exiting 13:56:29 (3296): No heartbeat from core client for 30 sec - exiting 13:56:30 (3296): No heartbeat from core client for 30 sec - exiting 13:56:31 (3296): No heartbeat from core client for 30 sec - exiting 13:56:32 (3296): No heartbeat from core client for 30 sec - exiting 13:56:33 (3296): No heartbeat from core client for 30 sec - exiting 13:56:34 (3296): No heartbeat from core client for 30 sec - exiting 13:56:36 (3296): No heartbeat from core client for 30 sec - exiting 13:56:37 (3296): No heartbeat from core client for 30 sec - exiting 13:56:38 (3296): No heartbeat from core client for 30 sec - exiting 13:56:39 (3296): No heartbeat from core client for 30 sec - exiting 13:56:40 (3296): No heartbeat from core client for 30 sec - exiting 13:56:41 (3296): No heartbeat from core client for 30 sec - exiting 13:56:42 (3296): No heartbeat from core client for 30 sec - exiting 13:56:43 (3296): No heartbeat from core client for 30 sec - exiting 13:56:44 (3296): No heartbeat from core client for 30 sec - exiting 13:56:45 (3296): No heartbeat from core client for 30 sec - exiting 13:56:46 (3296): No heartbeat from core client for 30 sec - exiting 13:56:48 (3296): No heartbeat from core client for 30 sec - exiting 13:56:49 (3296): No heartbeat from core client for 30 sec - exiting 13:56:50 (3296): No heartbeat from core client for 30 sec - exiting 13:56:51 (3296): No heartbeat from core client for 30 sec - exiting 13:56:52 (3296): No heartbeat from core client for 30 sec - exiting 13:56:53 (3296): No heartbeat from core client for 30 sec - exiting 13:56:54 (3296): No heartbeat from core client for 30 sec - exiting 13:56:55 (3296): No heartbeat from core client for 30 sec - exiting 13:56:58 (3296): No heartbeat from core client for 30 sec - exiting 13:56:59 (3296): No heartbeat from core client for 30 sec - exiting 13:57:00 (3296): No heartbeat from core client for 30 sec - exiting 13:57:01 (3296): No heartbeat from core client for 30 sec - exiting 13:57:02 (3296): No heartbeat from core client for 30 sec - exiting 13:57:03 (3296): No heartbeat from core client for 30 sec - exiting 13:57:04 (3296): No heartbeat from core client for 30 sec - exiting 13:57:05 (3296): No heartbeat from core client for 30 sec - exiting 13:57:06 (3296): No heartbeat from core client for 30 sec - exiting 13:57:07 (3296): No heartbeat from core client for 30 sec - exiting 13:57:09 (3296): No heartbeat from core client for 30 sec - exiting 13:57:10 (3296): No heartbeat from core client for 30 sec - exiting 13:57:11 (3296): No heartbeat from core client for 30 sec - exiting 13:57:12 (3296): No heartbeat from core client for 30 sec - exiting 13:57:13 (3296): No heartbeat from core client for 30 sec - exiting 13:57:14 (3296): No heartbeat from core client for 30 sec - exiting 13:57:15 (3296): No heartbeat from core client for 30 sec - exiting 13:57:16 (3296): No heartbeat from core client for 30 sec - exiting 13:57:17 (3296): No heartbeat from core client for 30 sec - exiting 13:57:18 (3296): No heartbeat from core client for 30 sec - exiting 13:57:19 (3296): No heartbeat from core client for 30 sec - exiting 13:57:21 (3296): No heartbeat from core client for 30 sec - exiting 13:57:22 (3296): No heartbeat from core client for 30 sec - exiting 13:57:23 (3296): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:42:36 (2008): No heartbeat from core client for 30 sec - exiting 13:42:37 (2008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:38 (2008): No heartbeat from core client for 30 sec - exiting 13:42:39 (2008): No heartbeat from core client for 30 sec - exiting 13:42:40 (2008): No heartbeat from core client for 30 sec - exiting 13:42:41 (2008): No heartbeat from core client for 30 sec - exiting 13:42:42 (2008): No heartbeat from core client for 30 sec - exiting 13:42:44 (2008): No heartbeat from core client for 30 sec - exiting 13:42:45 (2008): No heartbeat from core client for 30 sec - exiting 13:42:46 (2008): No heartbeat from core client for 30 sec - exiting 13:42:47 (2008): No heartbeat from core client for 30 sec - exiting 13:42:48 (2008): No heartbeat from core client for 30 sec - exiting 13:42:49 (2008): No heartbeat from core client for 30 sec - exiting 13:42:50 (2008): No heartbeat from core client for 30 sec - exiting 13:42:51 (2008): No heartbeat from core client for 30 sec - exiting 13:42:52 (2008): No heartbeat from core client for 30 sec - exiting 13:42:53 (2008): No heartbeat from core client for 30 sec - exiting 13:42:54 (2008): No heartbeat from core client for 30 sec - exiting 13:42:56 (2008): No heartbeat from core client for 30 sec - exiting 13:42:57 (2008): No heartbeat from core client for 30 sec - exiting 13:42:58 (2008): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jan 2014 15:12:52	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	492,480	443,138	0.8998
12 Jan 2014 17:08:29	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	466,560	418,216	0.8964
12 Jan 2014 08:33:25	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	440,640	394,076	0.8943
12 Jan 2014 00:46:30	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	414,720	369,532	0.8910
11 Jan 2014 17:08:56	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	388,800	345,473	0.8886
11 Jan 2014 09:47:55	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	362,880	321,178	0.8851
11 Jan 2014 02:24:54	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	336,960	297,018	0.8815
10 Jan 2014 22:17:38	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	311,040	273,056	0.8779
10 Jan 2014 03:10:47	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	285,120	250,870	0.8799
09 Jan 2014 19:53:24	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	259,200	228,776	0.8826
08 Jan 2014 22:29:12	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	233,280	204,166	0.8752
08 Jan 2014 15:46:48	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	207,360	181,391	0.8748
05 Jan 2014 23:26:01	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	181,440	158,437	0.8732
05 Jan 2014 15:31:49	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	155,520	134,938	0.8677
03 Jan 2014 17:40:10	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	129,600	112,252	0.8661
03 Jan 2014 11:01:14	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	103,680	89,785	0.8660
03 Jan 2014 04:19:38	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	77,760	67,167	0.8638
02 Jan 2014 21:37:58	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	51,840	44,630	0.8609
01 Jan 2014 21:32:24	1235271	16192272	hadcm3n_y8w3_1980_40_008391840_4	25,920	21,771	0.8399