Task 14283827

Name	hadcm3n_yhzt_1980_40_007832351_0
Workunit	7987463
Created	18 Mar 2012, 17:00:33 UTC
Sent	18 Mar 2012, 17:03:43 UTC
Report deadline	18 Jun 2012, 0:30:54 UTC
Received	23 Apr 2012, 12:55:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	952423
Run time	13 days 3 hours 58 min 56 sec
CPU time	12 days 17 hours 9 min 19 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:49:50 (2536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:05:12 (996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:54:56 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:49:24 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:14:04 (4836): No heartbeat from core client for 30 sec - exiting 07:14:05 (4836): No heartbeat from core client for 30 sec - exiting 07:14:06 (4836): No heartbeat from core client for 30 sec - exiting 07:14:07 (4836): No heartbeat from core client for 30 sec - exiting 07:14:09 (4836): No heartbeat from core client for 30 sec - exiting 07:14:10 (4836): No heartbeat from core client for 30 sec - exiting 07:14:11 (4836): No heartbeat from core client for 30 sec - exiting 07:14:12 (4836): No heartbeat from core client for 30 sec - exiting 07:14:13 (4836): No heartbeat from core client for 30 sec - exiting 07:14:14 (4836): No heartbeat from core client for 30 sec - exiting 07:14:15 (4836): No heartbeat from core client for 30 sec - exiting 07:14:16 (4836): No heartbeat from core client for 30 sec - exiting 07:14:17 (4836): No heartbeat from core client for 30 sec - exiting 07:14:18 (4836): No heartbeat from core client for 30 sec - exiting 07:14:19 (4836): No heartbeat from core client for 30 sec - exiting 07:14:21 (4836): No heartbeat from core client for 30 sec - exiting 07:14:22 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:25:41 (3412): No heartbeat from core client for 30 sec - exiting 07:25:42 (3412): No heartbeat from core client for 30 sec - exiting 07:25:43 (3412): No heartbeat from core client for 30 sec - exiting 07:25:44 (3412): No heartbeat from core client for 30 sec - exiting 07:25:45 (3412): No heartbeat from core client for 30 sec - exiting 07:25:46 (3412): No heartbeat from core client for 30 sec - exiting 07:25:47 (3412): No heartbeat from core client for 30 sec - exiting 07:25:48 (3412): No heartbeat from core client for 30 sec - exiting 07:25:49 (3412): No heartbeat from core client for 30 sec - exiting 07:25:50 (3412): No heartbeat from core client for 30 sec - exiting 07:25:51 (3412): No heartbeat from core client for 30 sec - exiting 07:25:52 (3412): No heartbeat from core client for 30 sec - exiting 07:25:53 (3412): No heartbeat from core client for 30 sec - exiting 07:25:57 (3412): No heartbeat from core client for 30 sec - exiting 07:25:58 (3412): No heartbeat from core client for 30 sec - exiting 07:25:59 (3412): No heartbeat from core client for 30 sec - exiting 07:26:00 (3412): No heartbeat from core client for 30 sec - exiting 07:26:01 (3412): No heartbeat from core client for 30 sec - exiting 07:26:02 (3412): No heartbeat from core client for 30 sec - exiting 07:26:03 (3412): No heartbeat from core client for 30 sec - exiting 07:26:04 (3412): No heartbeat from core client for 30 sec - exiting 07:26:05 (3412): No heartbeat from core client for 30 sec - exiting 07:26:06 (3412): No heartbeat from core client for 30 sec - exiting 07:26:07 (3412): No heartbeat from core client for 30 sec - exiting 07:26:08 (3412): No heartbeat from core client for 30 sec - exiting 07:26:09 (3412): No heartbeat from core client for 30 sec - exiting 07:26:10 (3412): No heartbeat from core client for 30 sec - exiting 07:26:11 (3412): No heartbeat from core client for 30 sec - exiting 07:26:12 (3412): No heartbeat from core client for 30 sec - exiting 07:26:13 (3412): No heartbeat from core client for 30 sec - exiting 07:26:14 (3412): No heartbeat from core client for 30 sec - exiting 07:26:15 (3412): No heartbeat from core client for 30 sec - exiting 07:26:16 (3412): No heartbeat from core client for 30 sec - exiting 07:26:17 (3412): No heartbeat from core client for 30 sec - exiting 07:26:18 (3412): No heartbeat from core client for 30 sec - exiting 07:26:19 (3412): No heartbeat from core client for 30 sec - exiting 07:26:20 (3412): No heartbeat from core client for 30 sec - exiting 07:26:21 (3412): No heartbeat from core client for 30 sec - exiting 07:26:22 (3412): No heartbeat from core client for 30 sec - exiting 07:26:23 (3412): No heartbeat from core client for 30 sec - exiting 07:26:24 (3412): No heartbeat from core client for 30 sec - exiting 07:26:25 (3412): No heartbeat from core client for 30 sec - exiting 07:26:26 (3412): No heartbeat from core client for 30 sec - exiting 07:26:27 (3412): No heartbeat from core client for 30 sec - exiting 07:26:28 (3412): No heartbeat from core client for 30 sec - exiting 07:26:29 (3412): No heartbeat from core client for 30 sec - exiting 07:26:30 (3412): No heartbeat from core client for 30 sec - exiting 07:26:31 (3412): No heartbeat from core client for 30 sec - exiting 07:26:32 (3412): No heartbeat from core client for 30 sec - exiting 07:26:33 (3412): No heartbeat from core client for 30 sec - exiting 07:26:34 (3412): No heartbeat from core client for 30 sec - exiting 07:26:35 (3412): No heartbeat from core client for 30 sec - exiting 07:26:36 (3412): No heartbeat from core client for 30 sec - exiting 07:26:37 (3412): No heartbeat from core client for 30 sec - exiting 07:26:38 (3412): No heartbeat from core client for 30 sec - exiting 07:26:39 (3412): No heartbeat from core client for 30 sec - exiting 07:26:40 (3412): No heartbeat from core client for 30 sec - exiting 07:26:41 (3412): No heartbeat from core client for 30 sec - exiting 07:26:42 (3412): No heartbeat from core client for 30 sec - exiting 07:26:43 (3412): No heartbeat from core client for 30 sec - exiting 07:26:44 (3412): No heartbeat from core client for 30 sec - exiting 07:26:46 (3412): No heartbeat from core client for 30 sec - exiting 07:26:47 (3412): No heartbeat from core client for 30 sec - exiting 07:26:48 (3412): No heartbeat from core client for 30 sec - exiting 07:26:49 (3412): No heartbeat from core client for 30 sec - exiting 07:26:50 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:51 (3412): No heartbeat from core client for 30 sec - exiting 07:26:52 (3412): No heartbeat from core client for 30 sec - exiting 07:26:53 (3412): No heartbeat from core client for 30 sec - exiting 07:26:54 (3412): No heartbeat from core client for 30 sec - exiting 07:26:55 (3412): No heartbeat from core client for 30 sec - exiting 07:26:56 (3412): No heartbeat from core client for 30 sec - exiting 07:26:57 (3412): No heartbeat from core client for 30 sec - exiting 07:26:58 (3412): No heartbeat from core client for 30 sec - exiting 07:26:59 (3412): No heartbeat from core client for 30 sec - exiting 07:27:00 (3412): No heartbeat from core client for 30 sec - exiting 08:01:44 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:45 (5768): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Apr 2012 20:36:07	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	648,000	1,083,097	1.6714
18 Apr 2012 18:07:16	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	622,080	1,038,399	1.6692
17 Apr 2012 20:57:20	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	596,160	995,322	1.6696
12 Apr 2012 14:20:31	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	570,240	950,772	1.6673
11 Apr 2012 19:52:07	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	544,320	909,348	1.6706
11 Apr 2012 01:48:51	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	518,400	867,731	1.6739
10 Apr 2012 08:41:20	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	492,480	826,378	1.6780
09 Apr 2012 13:55:14	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	466,560	784,016	1.6804
08 Apr 2012 16:07:04	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	440,640	742,119	1.6842
07 Apr 2012 12:46:25	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	414,720	698,229	1.6836
06 Apr 2012 14:37:31	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	388,800	654,462	1.6833
05 Apr 2012 19:00:08	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	362,880	613,433	1.6905
04 Apr 2012 22:24:49	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	336,960	570,998	1.6946
04 Apr 2012 04:47:49	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	311,040	529,789	1.7033
03 Apr 2012 09:30:06	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	285,120	488,754	1.7142
02 Apr 2012 13:23:12	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	259,200	447,034	1.7247
01 Apr 2012 10:26:17	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	233,280	403,304	1.7288
31 Mar 2012 05:21:24	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	207,360	361,540	1.7435
29 Mar 2012 20:42:01	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	181,440	319,902	1.7631
28 Mar 2012 08:01:38	952423	14283827	hadcm3n_yhzt_1980_40_007832351_0	155,520	271,402	1.7451