Task 12821940

Name	hadcm3n_p2n5_1900_40_007221009_0
Workunit	7419249
Created	26 Apr 2011, 15:21:46 UTC
Sent	2 May 2011, 4:05:00 UTC
Report deadline	1 Aug 2011, 11:32:11 UTC
Received	15 Jun 2011, 8:29:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1138353
Run time	12 days 19 hours 4 min 31 sec
CPU time	7 days 15 hours 56 min 2 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5220, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:56:26 (2608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:00:54 (3968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:55 (3968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1092, iMonCtr=1 Model crash detected, will tCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:54:07 (4056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7264, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... 14:56:03 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:13:49 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:03:46 (5508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:26:01 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:02 (4220): No heartbeat from core client for 30 sec - exiting 00:03:29 (5420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:14 (5000): No heartbeat from core client for 30 sec - exiting 00:05:15 (5000): No heartbeat from core client for 30 sec - exiting 00:05:16 (5000): No heartbeat from core client for 30 sec - exiting 00:05:17 (5000): No heartbeat from core client for 30 sec - exiting 00:05:18 (5000): No heartbeat from core client for 30 sec - exiting 00:05:19 (5000): No heartbeat from core client for 30 sec - exiting 00:05:20 (5000): No heartbeat from core client for 30 sec - exiting 00:05:22 (5000): No heartbeat from core client for 30 sec - exiting 00:05:23 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:24 (5000): No heartbeat from core client for 30 sec - exiting 00:05:25 (5000): No heartbeat from core client for 30 sec - exiting 00:05:26 (5000): No heartbeat from core client for 30 sec - exiting 12:44:47 (560): No heartbeat from core client for 30 sec - exiting 12:44:52 (560): No heartbeat from core client for 30 sec - exiting 12:44:54 (560): No heartbeat from core client for 30 sec - exiting 12:44:55 (560): No heartbeat from core client for 30 sec - exiting 12:44:56 (560): No heartbeat from core client for 30 sec - exiting 12:44:58 (560): No heartbeat from core client for 30 sec - exiting 12:44:59 (560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:40 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:16:25 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:16 (3020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:50:57 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:09:36 (4460): No heartbeat from core client for 30 sec - exiting 17:09:37 (4460): No heartbeat from core client for 30 sec - exiting 17:09:38 (4460): No heartbeat from core client for 30 sec - exiting 17:09:40 (4460): No heartbeat from core client for 30 sec - exiting 17:09:41 (4460): No heartbeat from core client for 30 sec - exiting 17:09:42 (4460): No heartbeat from core client for 30 sec - exiting 17:09:43 (4460): No heartbeat from core client for 30 sec - exiting 17:09:45 (4460): No heartbeat from core client for 30 sec - exiting 17:09:46 (4460): No heartbeat from core client for 30 sec - exiting 17:09:47 (4460): No heartbeat from core client for 30 sec - exiting 17:09:49 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:09:50 (4460): No heartbeat from core client for 30 sec - exiting 17:09:51 (4460): No heartbeat from core client for 30 sec - exiting 17:09:52 (4460): No heartbeat from core client for 30 sec - exiting 17:09:54 (4460): No heartbeat from core client for 30 sec - exiting 17:09:55 (4460): No heartbeat from core client for 30 sec - exiting 17:09:56 (4460): No heartbeat from core client for 30 sec - exiting 17:09:57 (4460): No heartbeat from core client for 30 sec - exiting 17:09:58 (4460): No heartbeat from core client for 30 sec - exiting 17:09:59 (4460): No heartbeat from core client for 30 sec - exiting 17:10:01 (4460): No heartbeat from core client for 30 sec - exiting 17:10:02 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:55:43 (424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:35 (3504): No heartbeat from core client for 30 sec - exiting 17:02:36 (3504): No heartbeat from core client for 30 sec - exiting 17:02:37 (3504): No heartbeat from core client for 30 sec - exiting 17:02:38 (3504): No heartbeat from core client for 30 sec - exiting 17:02:40 (3504): No heartbeat from core client for 30 sec - exiting 17:02:41 (3504): No heartbeat from core client for 30 sec - exiting 17:02:42 (3504): No heartbeat from core client for 30 sec - exiting 17:02:43 (3504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:44 (3504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:24:22 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5764, iMonCtr=1 Model crash detected, will try to restart... 19:06:39 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:03 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:39 (6840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:19:49 (4636): No heartbeat from core client for 30 sec - exiting 18:19:50 (4636): No heartbeat from core client for 30 sec - exiting 18:19:51 (4636): No heartbeat from core client for 30 sec - exiting 18:19:52 (4636): No heartbeat from core client for 30 sec - exiting 18:19:53 (4636): No heartbeat from core client for 30 sec - exiting 18:19:54 (4636): No heartbeat from core client for 30 sec - exiting 18:19:56 (4636): No heartbeat from core client for 30 sec - exiting 18:19:57 (4636): No heartbeat from core client for 30 sec - exiting 18:19:58 (4636): No heartbeat from core client for 30 sec - exiting 18:19:59 (4636): No heartbeat from core client for 30 sec - exiting 18:20:00 (4636): No heartbeat from core client for 30 sec - exiting 18:20:01 (4636): No heartbeat from core client for 30 sec - exiting 18:20:02 (4636): No heartbeat from core client for 30 sec - exiting 18:20:03 (4636): No heartbeat from core client for 30 sec - exiting 18:20:04 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:05 (4636): No heartbeat from core client for 30 sec - exiting 18:20:06 (4636): No heartbeat from core client for 30 sec - exiting 18:20:07 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7996, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5832, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:04:51 (4748): No heartbeat from core client for 30 sec - exiting 18:04:53 (4748): No heartbeat from core client for 30 sec - exiting 18:04:54 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5664, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6376, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Jun 2011 12:24:30	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	155,520	622,904	4.0053
01 Jun 2011 22:12:20	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	129,600	507,690	3.9174
22 May 2011 19:31:22	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	103,680	386,966	3.7323
18 May 2011 21:13:22	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	77,760	299,523	3.8519
08 May 2011 17:04:18	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	51,840	200,078	3.8595
06 May 2011 20:17:47	1138353	12821940	hadcm3n_p2n5_1900_40_007221009_0	25,920	106,414	4.1055