Task 16045806

Name	hadcm3n_ofet_1900_40_008475096_0
Workunit	8625935
Created	27 Sep 2013, 10:34:30 UTC
Sent	27 Sep 2013, 18:27:03 UTC
Report deadline	28 Dec 2013, 1:54:14 UTC
Received	10 Nov 2013, 9:02:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	995356
Run time	6 days 3 hours 1 min 5 sec
CPU time	4 days 23 hours 12 min 55 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	2.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:56:57 (672): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:05:02 (1776): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 18:50:33 (2508): No heartbeat from core client for 30 sec - exiting 18:50:34 (2508): No heartbeat from core client for 30 sec - exiting 18:50:35 (2508): No heartbeat from core client for 30 sec - exiting 18:50:36 (2508): No heartbeat from core client for 30 sec - exiting 18:50:37 (2508): No heartbeat from core client for 30 sec - exiting 18:50:38 (2508): No heartbeat from core client for 30 sec - exiting 18:50:39 (2508): No heartbeat from core client for 30 sec - exiting 18:50:40 (2508): No heartbeat from core client for 30 sec - exiting 18:50:41 (2508): No heartbeat from core client for 30 sec - exiting 18:50:42 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:06:42 (588): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:57:56 (2136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:43 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:52 (2464): No heartbeat from core client for 30 sec - exiting 08:05:54 (2464): No heartbeat from core client for 30 sec - exiting 08:05:55 (2464): No heartbeat from core client for 30 sec - exiting 08:05:56 (2464): No heartbeat from core client for 30 sec - exiting 08:05:57 (2464): No heartbeat from core client for 30 sec - exiting 08:05:58 (2464): No heartbeat from core client for 30 sec - exiting 08:05:59 (2464): No heartbeat from core client for 30 sec - exiting 08:06:00 (2464): No heartbeat from core client for 30 sec - exiting 08:06:01 (2464): No heartbeat from core client for 30 sec - exiting 08:06:02 (2464): No heartbeat from core client for 30 sec - exiting 08:06:03 (2464): No heartbeat from core client for 30 sec - exiting 08:06:04 (2464): No heartbeat from core client for 30 sec - exiting 08:06:05 (2464): No heartbeat from core client for 30 sec - exiting 08:06:06 (2464): No heartbeat from core client for 30 sec - exiting 08:06:07 (2464): No heartbeat from core client for 30 sec - exiting 08:06:08 (2464): No heartbeat from core client for 30 sec - exiting 08:06:09 (2464): No heartbeat from core client for 30 sec - exiting 08:06:10 (2464): No heartbeat from core client for 30 sec - exiting 08:06:11 (2464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:00 (1336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:02 (2336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:21 (584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:41 (2940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:30 (1584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:35 (1136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:56 (612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:17:28 (2472): No heartbeat from core client for 30 sec - exiting 10:17:29 (2472): No heartbeat from core client for 30 sec - exiting 10:17:30 (2472): No heartbeat from core client for 30 sec - exiting 10:17:31 (2472): No heartbeat from core client for 30 sec - exiting 10:17:32 (2472): No heartbeat from core client for 30 sec - exiting 10:17:33 (2472): No heartbeat from core client for 30 sec - exiting 10:17:34 (2472): No heartbeat from core client for 30 sec - exiting 10:17:35 (2472): No heartbeat from core client for 30 sec - exiting 10:17:36 (2472): No heartbeat from core client for 30 sec - exiting 10:17:37 (2472): No heartbeat from core client for 30 sec - exiting 10:17:38 (2472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2880, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Oct 2013 07:44:30	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	311,040	409,948	1.3180
06 Oct 2013 19:10:39	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	285,120	376,392	1.3201
06 Oct 2013 06:43:46	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	259,200	342,405	1.3210
05 Oct 2013 16:40:46	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	233,280	308,246	1.3214
04 Oct 2013 19:02:30	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	207,360	273,980	1.3213
04 Oct 2013 06:50:19	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	181,440	239,550	1.3203
03 Oct 2013 18:53:41	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	155,520	205,405	1.3208
03 Oct 2013 06:40:15	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	129,600	171,322	1.3219
02 Oct 2013 17:53:22	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	103,680	136,864	1.3201
30 Sep 2013 11:56:33	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	77,760	102,052	1.3124
30 Sep 2013 00:36:29	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	51,840	68,128	1.3142
29 Sep 2013 11:54:56	995356	16045806	hadcm3n_ofet_1900_40_008475096_0	25,920	33,955	1.3100