Task 15486299

Name	hadcm3n_3fyg_1940_40_008258719_0
Workunit	8413843
Created	20 Dec 2012, 11:54:35 UTC
Sent	20 Dec 2012, 11:54:56 UTC
Report deadline	21 Mar 2013, 19:22:07 UTC
Received	10 Jan 2013, 9:47:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1191909
Run time	6 days 8 hours 19 min 49 sec
CPU time	6 days 6 hours 46 min 42 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:48:43 (4600): No heartbeat from core client for 30 sec - exiting 14:48:44 (4600): No heartbeat from core client for 30 sec - exiting 14:48:45 (4600): No heartbeat from core client for 30 sec - exiting 14:48:46 (4600): No heartbeat from core client for 30 sec - exiting 14:48:47 (4600): No heartbeat from core client for 30 sec - exiting 14:48:48 (4600): No heartbeat from core client for 30 sec - exiting 14:48:49 (4600): No heartbeat from core client for 30 sec - exiting 14:48:50 (4600): No heartbeat from core client for 30 sec - exiting 14:48:52 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:45:33 (5896): No heartbeat from core client for 30 sec - exiting 09:45:34 (5896): No heartbeat from core client for 30 sec - exiting 09:45:35 (5896): No heartbeat from core client for 30 sec - exiting 09:45:36 (5896): No heartbeat from core client for 30 sec - exiting 09:45:37 (5896): No heartbeat from core client for 30 sec - exiting 09:45:38 (5896): No heartbeat from core client for 30 sec - exiting 09:45:39 (5896): No heartbeat from core client for 30 sec - exiting 09:45:40 (5896): No heartbeat from core client for 30 sec - exiting 09:45:41 (5896): No heartbeat from core client for 30 sec - exiting 09:45:43 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jan 2013 07:23:25	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	440,640	541,934	1.2299
09 Jan 2013 22:46:48	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	414,720	510,954	1.2320
09 Jan 2013 13:55:20	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	388,800	479,090	1.2322
09 Jan 2013 05:44:33	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	362,880	447,904	1.2343
08 Jan 2013 20:25:27	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	336,960	417,072	1.2377
08 Jan 2013 11:37:35	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	311,040	385,384	1.2390
08 Jan 2013 03:00:47	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	285,120	354,524	1.2434
07 Jan 2013 18:48:07	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	259,200	323,703	1.2489
07 Jan 2013 09:31:13	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	233,280	291,625	1.2501
04 Jan 2013 07:14:57	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	207,360	259,806	1.2529
03 Jan 2013 22:32:44	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	181,440	228,910	1.2616
03 Jan 2013 13:45:31	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	155,520	197,231	1.2682
03 Jan 2013 04:20:01	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	129,600	164,048	1.2658
02 Jan 2013 19:04:39	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	103,680	130,865	1.2622
02 Jan 2013 10:15:27	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	77,760	98,014	1.2605
21 Dec 2012 06:24:44	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	51,840	65,953	1.2722
20 Dec 2012 21:14:54	1191909	15486299	hadcm3n_3fyg_1940_40_008258719_0	25,920	33,067	1.2757