Task 15455063

Name	hadcm3n_zf8z_1880_40_008251758_0
Workunit	8406882
Created	22 Nov 2012, 19:16:25 UTC
Sent	22 Nov 2012, 19:16:35 UTC
Report deadline	22 Feb 2013, 2:43:46 UTC
Received	17 Dec 2012, 17:34:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1239578
Run time	15 days 7 hours 36 min 15 sec
CPU time	14 days 8 hours 19 min 42 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	3.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:05:02 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:57 (10184): No heartbeat from core client for 30 sec - exiting 19:03:59 (10184): No heartbeat from core client for 30 sec - exiting 19:04:00 (10184): No heartbeat from core client for 30 sec - exiting 19:04:01 (10184): No heartbeat from core client for 30 sec - exiting 19:04:02 (10184): No heartbeat from core client for 30 sec - exiting 19:04:03 (10184): No heartbeat from core client for 30 sec - exiting 19:04:04 (10184): No heartbeat from core client for 30 sec - exiting 19:04:05 (10184): No heartbeat from core client for 30 sec - exiting 19:04:06 (10184): No heartbeat from core client for 30 sec - exiting 19:04:07 (10184): No heartbeat from core client for 30 sec - exiting 19:04:08 (10184): No heartbeat from core client for 30 sec - exiting 19:04:09 (10184): No heartbeat from core client for 30 sec - exiting 19:04:10 (10184): No heartbeat from core client for 30 sec - exiting 19:04:11 (10184): No heartbeat from core client for 30 sec - exiting 19:04:12 (10184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5044, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Dec 2012 16:25:57	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	881,280	1,237,169	1.4038
17 Dec 2012 05:44:07	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	855,360	1,202,230	1.4055
16 Dec 2012 16:51:23	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	829,440	1,167,183	1.4072
16 Dec 2012 04:13:42	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	803,520	1,132,463	1.4094
15 Dec 2012 17:11:12	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	777,600	1,097,461	1.4113
15 Dec 2012 06:23:06	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	751,680	1,059,763	1.4099
14 Dec 2012 09:35:50	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	725,760	1,023,219	1.4099
14 Dec 2012 09:35:50	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	699,840	986,936	1.4102
14 Dec 2012 09:35:50	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	673,920	950,190	1.4099
14 Dec 2012 09:35:50	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	648,000	913,178	1.4092
06 Dec 2012 01:52:04	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	622,080	878,154	1.4116
05 Dec 2012 14:10:38	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	596,160	842,556	1.4133
04 Dec 2012 22:47:15	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	570,240	804,742	1.4112
04 Dec 2012 08:52:09	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	544,320	766,436	1.4081
03 Dec 2012 13:35:29	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	518,400	729,014	1.4063
02 Dec 2012 22:47:22	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	492,480	691,533	1.4042
02 Dec 2012 10:03:52	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	466,560	654,250	1.4023
01 Dec 2012 21:56:14	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	440,640	617,303	1.4009
01 Dec 2012 08:41:13	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	414,720	580,230	1.3991
30 Nov 2012 19:46:36	1239578	15455063	hadcm3n_zf8z_1880_40_008251758_0	388,800	542,973	1.3965