Task 14761722

Name	hadcm3n_o1z5_1980_40_007999162_0
Workunit	8154276
Created	31 May 2012, 17:40:05 UTC
Sent	31 May 2012, 17:41:23 UTC
Report deadline	31 Aug 2012, 1:08:34 UTC
Received	6 Jul 2012, 12:00:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1220473
Run time	27 days 14 hours 17 min 6 sec
CPU time	26 days 21 hours 5 min 51 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 12:27:11 (976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6424, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5268, iMonCtr=1 Model crash detected, will try to restart... 10:05:36 (3660): No heartbeat from core client for 30 sec - exiting 10:05:38 (3660): No heartbeat from core client for 30 sec - exiting 10:05:39 (3660): No heartbeat from core client for 30 sec - exiting 10:05:40 (3660): No heartbeat from core client for 30 sec - exiting 10:05:41 (3660): No heartbeat from core client for 30 sec - exiting 10:05:42 (3660): No heartbeat from core client for 30 sec - exiting 10:05:43 (3660): No heartbeat from core client for 30 sec - exiting 10:05:44 (3660): No heartbeat from core client for 30 sec - exiting 10:05:45 (3660): No heartbeat from core client for 30 sec - exiting 10:05:46 (3660): No heartbeat from core client for 30 sec - exiting 10:05:47 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:02 (6508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:22:03 (4088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:37:40 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:36:40 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:38 (8580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:34 (8612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:33:44 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:26 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3712, iMonCtr=1 Model crash detected, will try to restart... 14:02:40 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:37 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=480, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 00:25:55 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5868, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3496, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2012 02:30:39	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	596,160	2,221,150	3.7258
02 Jul 2012 17:32:04	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	570,240	2,108,317	3.6972
02 Jul 2012 13:32:50	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	544,320	1,996,947	3.6687
30 Jun 2012 01:05:34	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	518,400	1,890,462	3.6467
28 Jun 2012 15:55:38	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	492,480	1,777,963	3.6102
27 Jun 2012 07:52:04	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	466,560	1,665,438	3.5696
26 Jun 2012 07:34:27	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	440,640	1,552,912	3.5242
19 Jun 2012 09:06:19	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	414,720	1,440,707	3.4739
18 Jun 2012 00:07:37	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	388,800	1,329,008	3.4182
16 Jun 2012 20:48:22	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	362,880	1,236,169	3.4066
16 Jun 2012 01:47:43	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	336,960	1,168,056	3.4665
15 Jun 2012 06:38:24	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	311,040	1,100,062	3.5367
14 Jun 2012 11:19:27	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	285,120	1,031,534	3.6179
13 Jun 2012 18:02:38	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	259,200	966,702	3.7296
13 Jun 2012 00:53:07	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	233,280	908,652	3.8951
12 Jun 2012 10:13:19	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	207,360	856,411	4.1301
11 Jun 2012 11:17:42	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	181,440	774,960	4.2712
10 Jun 2012 04:18:13	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	155,520	663,716	4.2677
07 Jun 2012 17:48:13	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	129,600	553,453	4.2705
06 Jun 2012 03:09:04	1220473	14761722	hadcm3n_o1z5_1980_40_007999162_0	103,680	441,899	4.2621