Task 15927244

Name	hadcm3n_4j8o_2020_40_008396273_2
Workunit	8547132
Created	20 Aug 2013, 3:46:38 UTC
Sent	20 Aug 2013, 4:00:50 UTC
Report deadline	19 Nov 2013, 11:28:01 UTC
Received	5 Sep 2013, 5:02:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1099619
Run time	6 days 12 hours 0 min 25 sec
CPU time	6 days 0 hours 20 min 22 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:34:57 (8652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:58 (8652): No heartbeat from core client for 30 sec - exiting 22:34:59 (8652): No heartbeat from core client for 30 sec - exiting 22:35:00 (8652): No heartbeat from core client for 30 sec - exiting 22:35:01 (8652): No heartbeat from core client for 30 sec - exiting 22:35:02 (8652): No heartbeat from core client for 30 sec - exiting 22:35:03 (8652): No heartbeat from core client for 30 sec - exiting 22:35:04 (8652): No heartbeat from core client for 30 sec - exiting 22:35:05 (8652): No heartbeat from core client for 30 sec - exiting 22:35:06 (8652): No heartbeat from core client for 30 sec - exiting 00:45:54 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:49:01 (10460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:02 (10460): No heartbeat from core client for 30 sec - exiting 08:49:03 (10460): No heartbeat from core client for 30 sec - exiting 08:49:04 (10460): No heartbeat from core client for 30 sec - exiting 08:49:05 (10460): No heartbeat from core client for 30 sec - exiting 08:49:06 (10460): No heartbeat from core client for 30 sec - exiting 08:49:07 (10460): No heartbeat from core client for 30 sec - exiting 08:49:08 (10460): No heartbeat from core client for 30 sec - exiting 08:49:09 (10460): No heartbeat from core client for 30 sec - exiting 08:49:10 (10460): No heartbeat from core client for 30 sec - exiting 08:49:11 (10460): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:10:20 (9188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:21 (9188): No heartbeat from core client for 30 sec - exiting 22:10:22 (9188): No heartbeat from core client for 30 sec - exiting 22:10:23 (9188): No heartbeat from core client for 30 sec - exiting 22:10:24 (9188): No heartbeat from core client for 30 sec - exiting 22:10:25 (9188): No heartbeat from core client for 30 sec - exiting 22:10:26 (9188): No heartbeat from core client for 30 sec - exiting 22:10:27 (9188): No heartbeat from core client for 30 sec - exiting 22:10:28 (9188): No heartbeat from core client for 30 sec - exiting 22:10:29 (9188): No heartbeat from core client for 30 sec - exiting 22:10:30 (9188): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 02:01:17 (8212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:18 (8212): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 04:07:34 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:13 (9948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:15 (9948): No heartbeat from core client for 30 sec - exiting 08:01:16 (9948): No heartbeat from core client for 30 sec - exiting 08:01:17 (9948): No heartbeat from core client for 30 sec - exiting 08:01:18 (9948): No heartbeat from core client for 30 sec - exiting 08:01:19 (9948): No heartbeat from core client for 30 sec - exiting 08:01:20 (9948): No heartbeat from core client for 30 sec - exiting 08:01:21 (9948): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:25:22 (7552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3964, iMonCtr=1 Model crash detected, will try to restart... 08:45:36 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:54:19 (10652): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:17:55 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10448, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8692, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Sep 2013 14:01:19	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	285,120	513,031	1.7994
31 Aug 2013 11:46:11	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	259,200	472,978	1.8248
30 Aug 2013 08:58:04	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	233,280	427,467	1.8324
29 Aug 2013 07:13:41	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	207,360	380,344	1.8342
28 Aug 2013 05:47:12	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	181,440	333,168	1.8362
26 Aug 2013 16:05:57	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	155,520	285,528	1.8360
25 Aug 2013 14:29:16	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	129,600	237,920	1.8358
24 Aug 2013 12:34:07	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	103,680	190,475	1.8371
23 Aug 2013 10:06:23	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	77,760	143,081	1.8400
22 Aug 2013 08:24:27	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	51,840	95,510	1.8424
21 Aug 2013 06:41:17	1099619	15927244	hadcm3n_4j8o_2020_40_008396273_2	25,920	47,654	1.8385