Task 13824060

Name	hadcm3n_yedd_1940_40_007547821_4
Workunit	7745053
Created	27 Dec 2011, 16:47:23 UTC
Sent	27 Dec 2011, 16:47:29 UTC
Report deadline	28 Mar 2012, 0:14:40 UTC
Received	29 Jan 2012, 22:54:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1086810
Run time	21 days 7 hours 46 min 35 sec
CPU time	20 days 19 hours 47 min 18 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.57 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 10:14:20 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:25:31 (6056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:25:32 (6056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 14:36:38 (6216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:08:06 (8820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:08:08 (8820): No heartbeat from core client for 30 sec - exiting 00:08:09 (8820): No heartbeat from core client for 30 sec - exiting 00:08:10 (8820): No heartbeat from core client for 30 sec - exiting 00:08:11 (8820): No heartbeat from core client for 30 sec - exiting 00:08:12 (8820): No heartbeat from core client for 30 sec - exiting 00:08:13 (8820): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 11:59:13 (10684): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:53:41 (3000): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:12:33 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:15:38 (5672): No heartbeat from core client for 30 sec - exiting 12:15:39 (5672): No heartbeat from core client for 30 sec - exiting 12:15:40 (5672): No heartbeat from core client for 30 sec - exiting 12:15:42 (5672): No heartbeat from core client for 30 sec - exiting 12:15:43 (5672): No heartbeat from core client for 30 sec - exiting 12:15:44 (5672): No heartbeat from core client for 30 sec - exiting 12:15:45 (5672): No heartbeat from core client for 30 sec - exiting 12:15:46 (5672): No heartbeat from core client for 30 sec - exiting 12:15:47 (5672): No heartbeat from core client for 30 sec - exiting 12:15:48 (5672): No heartbeat from core client for 30 sec - exiting 12:15:49 (5672): No heartbeat from core client for 30 sec - exiting 12:15:50 (5672): No heartbeat from core client for 30 sec - exiting 12:15:51 (5672): No heartbeat from core client for 30 sec - exiting 12:15:53 (5672): No heartbeat from core client for 30 sec - exiting 12:15:54 (5672): No heartbeat from core client for 30 sec - exiting 12:15:55 (5672): No heartbeat from core client for 30 sec - exiting 12:15:56 (5672): No heartbeat from core client for 30 sec - exiting 12:15:57 (5672): No heartbeat from core client for 30 sec - exiting 12:15:58 (5672): No heartbeat from core client for 30 sec - exiting 12:15:59 (5672): No heartbeat from core client for 30 sec - exiting 12:16:00 (5672): No heartbeat from core client for 30 sec - exiting 12:16:01 (5672): No heartbeat from core client for 30 sec - exiting 12:16:02 (5672): No heartbeat from core client for 30 sec - exiting 12:16:03 (5672): No heartbeat from core client for 30 sec - exiting 12:16:05 (5672): No heartbeat from core client for 30 sec - exiting 12:16:06 (5672): No heartbeat from core client for 30 sec - exiting 12:16:07 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:08 (5672): No heartbeat from core client for 30 sec - exiting 12:16:09 (5672): No heartbeat from core client for 30 sec - exiting 12:16:10 (5672): No heartbeat from core client for 30 sec - exiting 12:16:11 (5672): No heartbeat from core client for 30 sec - exiting 12:16:12 (5672): No heartbeat from core client for 30 sec - exiting 12:16:13 (5672): No heartbeat from core client for 30 sec - exiting 12:16:14 (5672): No heartbeat from core client for 30 sec - exiting 12:16:15 (5672): No heartbeat from core client for 30 sec - exiting 12:16:17 (5672): No heartbeat from core client for 30 sec - exiting 12:16:18 (5672): No heartbeat from core client for 30 sec - exiting 12:20:16 (3432): No heartbeat from core client for 30 sec - exiting 12:20:17 (3432): No heartbeat from core client for 30 sec - exiting 12:20:18 (3432): No heartbeat from core client for 30 sec - exiting 12:20:19 (3432): No heartbeat from core client for 30 sec - exiting 12:20:20 (3432): No heartbeat from core client for 30 sec - exiting 12:20:21 (3432): No heartbeat from core client for 30 sec - exiting 12:20:22 (3432): No heartbeat from core client for 30 sec - exiting 12:20:23 (3432): No heartbeat from core client for 30 sec - exiting 12:20:24 (3432): No heartbeat from core client for 30 sec - exiting 12:20:25 (3432): No heartbeat from core client for 30 sec - exiting 12:20:27 (3432): No heartbeat from core client for 30 sec - exiting 12:20:28 (3432): No heartbeat from core client for 30 sec - exiting 12:20:29 (3432): No heartbeat from core client for 30 sec - exiting 12:20:30 (3432): No heartbeat from core client for 30 sec - exiting 12:20:31 (3432): No heartbeat from core client for 30 sec - exiting 12:20:32 (3432): No heartbeat from core client for 30 sec - exiting 12:20:33 (3432): No heartbeat from core client for 30 sec - exiting 12:20:34 (3432): No heartbeat from core client for 30 sec - exiting 12:20:35 (3432): No heartbeat from core client for 30 sec - exiting 12:20:36 (3432): No heartbeat from core client for 30 sec - exiting 12:20:37 (3432): No heartbeat from core client for 30 sec - exiting 12:20:39 (3432): No heartbeat from core client for 30 sec - exiting 12:20:40 (3432): No heartbeat from core client for 30 sec - exiting 12:20:41 (3432): No heartbeat from core client for 30 sec - exiting 12:20:42 (3432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:01 (924): No heartbeat from core client for 30 sec - exiting 13:44:02 (924): No heartbeat from core client for 30 sec - exiting 13:44:03 (924): No heartbeat from core client for 30 sec - exiting 13:44:04 (924): No heartbeat from core client for 30 sec - exiting 13:44:05 (924): No heartbeat from core client for 30 sec - exiting 13:44:06 (924): No heartbeat from core client for 30 sec - exiting 13:44:07 (924): No heartbeat from core client for 30 sec - exiting 13:44:08 (924): No heartbeat from core client for 30 sec - exiting 13:44:09 (924): No heartbeat from core client for 30 sec - exiting 13:44:11 (924): No heartbeat from core client for 30 sec - exiting 13:44:12 (924): No heartbeat from core client for 30 sec - exiting 13:44:13 (924): No heartbeat from core client for 30 sec - exiting 13:44:14 (924): No heartbeat from core client for 30 sec - exiting 13:44:15 (924): No heartbeat from core client for 30 sec - exiting 13:44:16 (924): No heartbeat from core client for 30 sec - exiting 13:44:17 (924): No heartbeat from core client for 30 sec - exiting 13:44:18 (924): No heartbeat from core client for 30 sec - exiting 13:44:19 (924): No heartbeat from core client for 30 sec - exiting 13:44:20 (924): No heartbeat from core client for 30 sec - exiting 13:44:21 (924): No heartbeat from core client for 30 sec - exiting 13:44:23 (924): No heartbeat from core client for 30 sec - exiting 13:44:24 (924): No heartbeat from core client for 30 sec - exiting 13:44:25 (924): No heartbeat from core client for 30 sec - exiting 13:44:26 (924): No heartbeat from core client for 30 sec - exiting 13:44:27 (924): No heartbeat from core client for 30 sec - exiting 13:44:28 (924): No heartbeat from core client for 30 sec - exiting 13:44:29 (924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 15:07:51 (3256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:38:51 (4680): No heartbeat from core client for 30 sec - exiting 15:38:52 (4680): No heartbeat from core client for 30 sec - exiting 15:38:53 (4680): No heartbeat from core client for 30 sec - exiting 15:38:54 (4680): No heartbeat from core client for 30 sec - exiting 15:38:55 (4680): No heartbeat from core client for 30 sec - exiting 15:38:56 (4680): No heartbeat from core client for 30 sec - exiting 15:38:57 (4680): No heartbeat from core client for 30 sec - exiting 15:38:58 (4680): No heartbeat from core client for 30 sec - exiting 15:38:59 (4680): No heartbeat from core client for 30 sec - exiting 15:39:00 (4680): No heartbeat from core client for 30 sec - exiting 15:39:01 (4680): No heartbeat from core client for 30 sec - exiting 15:39:02 (4680): No heartbeat from core client for 30 sec - exiting 15:39:03 (4680): No heartbeat from core client for 30 sec - exiting 15:39:04 (4680): No heartbeat from core client for 30 sec - exiting 15:39:05 (4680): No heartbeat from core client for 30 sec - exiting 15:39:06 (4680): No heartbeat from core client for 30 sec - exiting 15:39:07 (4680): No heartbeat from core client for 30 sec - exiting 15:39:08 (4680): No heartbeat from core client for 30 sec - exiting 15:39:09 (4680): No heartbeat from core client for 30 sec - exiting 15:39:10 (4680): No heartbeat from core client for 30 sec - exiting 15:39:11 (4680): No heartbeat from core client for 30 sec - exiting 15:39:12 (4680): No heartbeat from core client for 30 sec - exiting 15:39:13 (4680): No heartbeat from core client for 30 sec - exiting 15:39:14 (4680): No heartbeat from core client for 30 sec - exiting 15:39:15 (4680): No heartbeat from core client for 30 sec - exiting 15:39:16 (4680): No heartbeat from core client for 30 sec - exiting 15:39:17 (4680): No heartbeat from core client for 30 sec - exiting 15:39:18 (4680): No heartbeat from core client for 30 sec - exiting 15:39:19 (4680): No heartbeat from core client for 30 sec - exiting 15:39:20 (4680): No heartbeat from core client for 30 sec - exiting 15:39:21 (4680): No heartbeat from core client for 30 sec - exiting 15:39:22 (4680): No heartbeat from core client for 30 sec - exiting 15:39:23 (4680): No heartbeat from core client for 30 sec - exiting 15:39:24 (4680): No heartbeat from core client for 30 sec - exiting 15:39:25 (4680): No heartbeat from core client for 30 sec - exiting 15:39:26 (4680): No heartbeat from core client for 30 sec - exiting 15:39:27 (4680): No heartbeat from core client for 30 sec - exiting 15:39:28 (4680): No heartbeat from core client for 30 sec - exiting 15:39:29 (4680): No heartbeat from core client for 30 sec - exiting 15:39:30 (4680): No heartbeat from core client for 30 sec - exiting 15:39:31 (4680): No heartbeat from core client for 30 sec - exiting 15:39:32 (4680): No heartbeat from core client for 30 sec - exiting 15:39:33 (4680): No heartbeat from core client for 30 sec - exiting 15:39:34 (4680): No heartbeat from core client for 30 sec - exiting 15:39:35 (4680): No heartbeat from core client for 30 sec - exiting 15:39:36 (4680): No heartbeat from core client for 30 sec - exiting 15:39:37 (4680): No heartbeat from core client for 30 sec - exiting 15:39:38 (4680): No heartbeat from core client for 30 sec - exiting 15:39:39 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:45:17 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:10 (3688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:25:17 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:26:01 (8064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:26:35 (5640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:11 (7860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:58 (832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 22:48:44 (3616): No heartbeat from core client for 30 sec - exiting 22:48:46 (3616): No heartbeat from core client for 30 sec - exiting 22:48:47 (3616): No heartbeat from core client for 30 sec - exiting 22:48:48 (3616): No heartbeat from core client for 30 sec - exiting 22:48:49 (3616): No heartbeat from core client for 30 sec - exiting 22:48:50 (3616): No heartbeat from core client for 30 sec - exiting 22:48:51 (3616): No heartbeat from core client for 30 sec - exiting 22:48:52 (3616): No heartbeat from core client for 30 sec - exiting 22:48:53 (3616): No heartbeat from core client for 30 sec - exiting 22:48:54 (3616): No heartbeat from core client for 30 sec - exiting 22:48:55 (3616): No heartbeat from core client for 30 sec - exiting 22:48:56 (3616): No heartbeat from core client for 30 sec - exiting 22:48:57 (3616): No heartbeat from core client for 30 sec - exiting 22:48:58 (3616): No heartbeat from core client for 30 sec - exiting 22:48:59 (3616): No heartbeat from core client for 30 sec - exiting 22:49:00 (3616): No heartbeat from core client for 30 sec - exiting 22:49:01 (3616): No heartbeat from core client for 30 sec - exiting 22:49:02 (3616): No heartbeat from core client for 30 sec - exiting 22:49:03 (3616): No heartbeat from core client for 30 sec - exiting 22:49:04 (3616): No heartbeat from core client for 30 sec - exiting 22:49:05 (3616): No heartbeat from core client for 30 sec - exiting 22:49:06 (3616): No heartbeat from core client for 30 sec - exiting 22:49:07 (3616): No heartbeat from core client for 30 sec - exiting 22:49:08 (3616): No heartbeat from core client for 30 sec - exiting 22:49:09 (3616): No heartbeat from core client for 30 sec - exiting 22:49:10 (3616): No heartbeat from core client for 30 sec - exiting 22:49:11 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:12 (3616): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2336, iMonCtr=1 Model crash detected, will try to restart... 22:49:34 (5256): Can't acquire lockfile (32) - waiting 35s Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2336, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2336, iMonCtr=1 Model crash detected, will try to restart... 22:49:54 (2336): No heartbeat from core client for 30 sec - exiting 22:49:55 (2336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5256, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5256, iMonCtr=1 Model crash detected, will try to restart... 22:51:08 (5256): No heartbeat from core client for 30 sec - exiting 22:51:09 (5256): No heartbeat from core client for 30 sec - exiting 22:51:10 (5256): No heartbeat from core client for 30 sec - exiting 22:51:11 (5256): No heartbeat from core client for 30 sec - exiting 22:51:12 (5256): No heartbeat from core client for 30 sec - exiting 22:51:13 (5256): No heartbeat from core client for 30 sec - exiting 22:51:14 (5256): No heartbeat from core client for 30 sec - exiting 22:51:15 (5256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1156, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jan 2012 12:06:29	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	959,040	1,764,869	1.8402
29 Jan 2012 01:50:18	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	933,120	1,727,383	1.8512
28 Jan 2012 13:28:53	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	907,200	1,689,678	1.8625
28 Jan 2012 02:57:21	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	881,280	1,652,119	1.8747
27 Jan 2012 16:17:57	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	855,360	1,614,563	1.8876
27 Jan 2012 05:39:34	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	829,440	1,577,229	1.9016
26 Jan 2012 19:12:30	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	803,520	1,539,663	1.9161
26 Jan 2012 03:21:48	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	777,600	1,496,885	1.9250
25 Jan 2012 14:01:29	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	751,680	1,450,333	1.9295
25 Jan 2012 00:37:59	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	725,760	1,403,564	1.9339
24 Jan 2012 11:05:57	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	699,840	1,355,646	1.9371
23 Jan 2012 22:38:37	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	673,920	1,307,919	1.9408
23 Jan 2012 07:55:00	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	648,000	1,259,727	1.9440
17 Jan 2012 08:28:05	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	622,080	1,210,979	1.9467
16 Jan 2012 18:12:40	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	596,160	1,161,515	1.9483
16 Jan 2012 04:13:37	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	570,240	1,111,900	1.9499
15 Jan 2012 11:56:04	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	544,320	1,061,734	1.9506
14 Jan 2012 20:22:23	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	518,400	1,011,032	1.9503
14 Jan 2012 02:20:48	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	492,480	958,487	1.9462
13 Jan 2012 09:22:47	1086810	13824060	hadcm3n_yedd_1940_40_007547821_4	466,560	908,473	1.9472