Task 15587894

Name	hadcm3n_4ek1_1940_40_008302509_0
Workunit	8453644
Created	6 Feb 2013, 19:19:59 UTC
Sent	6 Feb 2013, 19:20:13 UTC
Report deadline	9 May 2013, 2:47:24 UTC
Received	20 Mar 2013, 9:18:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1233640
Run time	19 days 18 hours 59 min 4 sec
CPU time	16 days 7 hours 27 min 58 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:09:31 (4748): No heartbeat from core client for 30 sec - exiting 07:09:32 (4748): No heartbeat from core client for 30 sec - exiting 07:09:33 (4748): No heartbeat from core client for 30 sec - exiting 07:09:34 (4748): No heartbeat from core client for 30 sec - exiting 07:09:35 (4748): No heartbeat from core client for 30 sec - exiting 07:09:36 (4748): No heartbeat from core client for 30 sec - exiting 07:09:37 (4748): No heartbeat from core client for 30 sec - exiting 07:09:38 (4748): No heartbeat from core client for 30 sec - exiting 07:09:39 (4748): No heartbeat from core client for 30 sec - exiting 07:09:40 (4748): No heartbeat from core client for 30 sec - exiting 07:09:41 (4748): No heartbeat from core client for 30 sec - exiting 07:09:42 (4748): No heartbeat from core client for 30 sec - exiting 07:09:43 (4748): No heartbeat from core client for 30 sec - exiting 07:09:44 (4748): No heartbeat from core client for 30 sec - exiting 07:09:45 (4748): No heartbeat from core client for 30 sec - exiting 07:09:46 (4748): No heartbeat from core client for 30 sec - exiting 07:09:47 (4748): No heartbeat from core client for 30 sec - exiting 07:09:48 (4748): No heartbeat from core client for 30 sec - exiting 07:09:49 (4748): No heartbeat from core client for 30 sec - exiting 07:09:50 (4748): No heartbeat from core client for 30 sec - exiting 07:09:52 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2812, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:50:41 (4736): No heartbeat from core client for 30 sec - exiting 07:50:42 (4736): No heartbeat from core client for 30 sec - exiting 07:50:43 (4736): No heartbeat from core client for 30 sec - exiting 07:50:44 (4736): No heartbeat from core client for 30 sec - exiting 07:50:45 (4736): No heartbeat from core client for 30 sec - exiting 07:50:46 (4736): No heartbeat from core client for 30 sec - exiting 07:50:47 (4736): No heartbeat from core client for 30 sec - exiting 07:50:48 (4736): No heartbeat from core client for 30 sec - exiting 07:50:49 (4736): No heartbeat from core client for 30 sec - exiting 07:50:50 (4736): No heartbeat from core client for 30 sec - exiting 07:50:52 (4736): No heartbeat from core client for 30 sec - exiting 07:50:53 (4736): No heartbeat from core client for 30 sec - exiting 07:50:54 (4736): No heartbeat from core client for 30 sec - exiting 07:50:55 (4736): No heartbeat from core client for 30 sec - exiting 07:50:56 (4736): No heartbeat from core client for 30 sec - exiting 07:50:57 (4736): No heartbeat from core client for 30 sec - exiting 07:50:58 (4736): No heartbeat from core client for 30 sec - exiting 07:50:59 (4736): No heartbeat from core client for 30 sec - exiting 07:51:00 (4736): No heartbeat from core client for 30 sec - exiting 07:51:01 (4736): No heartbeat from core client for 30 sec - exiting 07:51:03 (4736): No heartbeat from core client for 30 sec - exiting 07:51:04 (4736): No heartbeat from core client for 30 sec - exiting 07:51:05 (4736): No heartbeat from core client for 30 sec - exiting 07:51:06 (4736): No heartbeat from core client for 30 sec - exiting 07:51:07 (4736): No heartbeat from core client for 30 sec - exiting 07:51:08 (4736): No heartbeat from core client for 30 sec - exiting 07:51:09 (4736): No heartbeat from core client for 30 sec - exiting 07:51:10 (4736): No heartbeat from core client for 30 sec - exiting 07:51:11 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2664, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7632, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1712, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... 23:13:36 (3876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10656, iMonCtr=1 Model crash detected, will try to restart... 07:22:13 (940): No heartbeat from core client for 30 sec - exiting 07:22:14 (940): No heartbeat from core client for 30 sec - exiting 07:22:15 (940): No heartbeat from core client for 30 sec - exiting 07:22:16 (940): No heartbeat from core client for 30 sec - exiting 07:22:17 (940): No heartbeat from core client for 30 sec - exiting 07:22:18 (940): No heartbeat from core client for 30 sec - exiting 07:22:19 (940): No heartbeat from core client for 30 sec - exiting 07:22:20 (940): No heartbeat from core client for 30 sec - exiting 07:22:21 (940): No heartbeat from core client for 30 sec - exiting 07:22:22 (940): No heartbeat from core client for 30 sec - exiting 07:22:23 (940): No heartbeat from core client for 30 sec - exiting 07:22:24 (940): No heartbeat from core client for 30 sec - exiting 07:22:25 (940): No heartbeat from core client for 30 sec - exiting 07:22:26 (940): No heartbeat from core client for 30 sec - exiting 07:22:27 (940): No heartbeat from core client for 30 sec - exiting 07:22:28 (940): No heartbeat from core client for 30 sec - exiting 07:22:29 (940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:44:32 (4844): No heartbeat from core client for 30 sec - exiting 06:44:33 (4844): No heartbeat from core client for 30 sec - exiting 06:44:34 (4844): No heartbeat from core client for 30 sec - exiting 06:44:35 (4844): No heartbeat from core client for 30 sec - exiting 06:44:36 (4844): No heartbeat from core client for 30 sec - exiting 06:44:38 (4844): No heartbeat from core client for 30 sec - exiting 06:44:39 (4844): No heartbeat from core client for 30 sec - exiting 06:44:40 (4844): No heartbeat from core client for 30 sec - exiting 06:44:41 (4844): No heartbeat from core client for 30 sec - exiting 06:44:42 (4844): No heartbeat from core client for 30 sec - exiting 06:44:43 (4844): No heartbeat from core client for 30 sec - exiting 06:44:44 (4844): No heartbeat from core client for 30 sec - exiting 06:44:45 (4844): No heartbeat from core client for 30 sec - exiting 06:44:46 (4844): No heartbeat from core client for 30 sec - exiting 06:44:47 (4844): No heartbeat from core client for 30 sec - exiting 06:44:48 (4844): No heartbeat from core client for 30 sec - exiting 06:44:49 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6928, iMonCtr=1 Model crash detected, will try to restart... 21:47:55 (4780): No heartbeat from core client for 30 sec - exiting 21:47:56 (4780): No heartbeat from core client for 30 sec - exiting 21:47:57 (4780): No heartbeat from core client for 30 sec - exiting 21:47:58 (4780): No heartbeat from core client for 30 sec - exiting 21:48:00 (4780): No heartbeat from core client for 30 sec - exiting 21:48:01 (4780): No heartbeat from core client for 30 sec - exiting 21:48:02 (4780): No heartbeat from core client for 30 sec - exiting 21:48:03 (4780): No heartbeat from core client for 30 sec - exiting 21:48:04 (4780): No heartbeat from core client for 30 sec - exiting 21:48:05 (4780): No heartbeat from core client for 30 sec - exiting 21:48:06 (4780): No heartbeat from core client for 30 sec - exiting 21:48:07 (4780): No heartbeat from core client for 30 sec - exiting 21:48:08 (4780): No heartbeat from core client for 30 sec - exiting 21:48:09 (4780): No heartbeat from core client for 30 sec - exiting 21:48:11 (4780): No heartbeat from core client for 30 sec - exiting 21:48:12 (4780): No heartbeat from core client for 30 sec - exiting 21:48:13 (4780): No heartbeat from core client for 30 sec - exiting 21:48:14 (4780): No heartbeat from core client for 30 sec - exiting 21:48:15 (4780): No heartbeat from core client for 30 sec - exiting 21:48:16 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Mar 2013 08:21:57	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	777,600	1,409,274	1.8123
18 Mar 2013 23:11:38	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	751,680	1,362,006	1.8119
13 Mar 2013 23:27:02	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	725,760	1,314,309	1.8109
12 Mar 2013 10:27:00	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	699,840	1,267,384	1.8110
11 Mar 2013 12:04:17	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	673,920	1,220,384	1.8109
10 Mar 2013 20:42:55	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	648,000	1,173,234	1.8105
10 Mar 2013 04:15:30	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	622,080	1,126,340	1.8106
09 Mar 2013 12:43:09	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	596,160	1,079,066	1.8100
08 Mar 2013 10:06:51	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	570,240	1,031,808	1.8094
07 Mar 2013 17:40:02	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	544,320	984,410	1.8085
06 Mar 2013 07:45:34	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	518,400	937,161	1.8078
05 Mar 2013 06:20:11	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	492,480	889,867	1.8069
01 Mar 2013 14:45:44	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	466,560	842,467	1.8057
28 Feb 2013 07:22:24	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	440,640	794,351	1.8027
27 Feb 2013 01:40:10	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	414,720	745,643	1.7979
25 Feb 2013 23:01:47	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	388,800	698,509	1.7966
21 Feb 2013 21:59:20	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	362,880	651,194	1.7945
21 Feb 2013 00:07:43	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	336,960	604,224	1.7932
20 Feb 2013 06:57:43	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	311,040	557,402	1.7921
19 Feb 2013 11:08:46	1233640	15587894	hadcm3n_4ek1_1940_40_008302509_0	285,120	510,173	1.7893