Task 14761178

Name	hadcm3n_o1uk_2020_40_007999003_0
Workunit	8154117
Created	31 May 2012, 16:16:40 UTC
Sent	31 May 2012, 16:16:46 UTC
Report deadline	30 Aug 2012, 23:43:57 UTC
Received	23 Jun 2012, 22:57:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1036046
Run time	5 days 17 hours 10 min 54 sec
CPU time	4 days 22 hours 52 min 55 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 01:21:47 (9996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:08 (3380): No heartbeat from core client for 30 sec - exiting 06:23:09 (3380): No heartbeat from core client for 30 sec - exiting 06:23:10 (3380): No heartbeat from core client for 30 sec - exiting 06:23:11 (3380): No heartbeat from core client for 30 sec - exiting 06:23:12 (3380): No heartbeat from core client for 30 sec - exiting 06:23:14 (3380): No heartbeat from core client for 30 sec - exiting 06:23:15 (3380): No heartbeat from core client for 30 sec - exiting 06:23:16 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4468, iMonCtr=1 Model crash detected, will try to restart... 14:25:29 (4588): No heartbeat from core client for 30 sec - exiting 14:25:30 (4588): No heartbeat from core client for 30 sec - exiting 14:25:31 (4588): No heartbeat from core client for 30 sec - exiting 14:25:32 (4588): No heartbeat from core client for 30 sec - exiting 14:25:33 (4588): No heartbeat from core client for 30 sec - exiting 14:25:34 (4588): No heartbeat from core client for 30 sec - exiting 14:25:35 (4588): No heartbeat from core client for 30 sec - exiting 14:25:37 (4588): No heartbeat from core client for 30 sec - exiting 14:25:38 (4588): No heartbeat from core client for 30 sec - exiting 14:25:39 (4588): No heartbeat from core client for 30 sec - exiting 14:25:40 (4588): No heartbeat from core client for 30 sec - exiting 14:25:41 (4588): No heartbeat from core client for 30 sec - exiting 14:25:42 (4588): No heartbeat from core client for 30 sec - exiting 14:25:43 (4588): No heartbeat from core client for 30 sec - exiting 14:25:44 (4588): No heartbeat from core client for 30 sec - exiting 14:25:45 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:25 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:28 (6044): No heartbeat from core client for 30 sec - exiting 14:41:29 (6044): No heartbeat from core client for 30 sec - exiting 14:41:30 (6044): No heartbeat from core client for 30 sec - exiting 14:41:31 (6044): No heartbeat from core client for 30 sec - exiting 14:41:32 (6044): No heartbeat from core client for 30 sec - exiting 14:41:33 (6044): No heartbeat from core client for 30 sec - exiting 14:41:34 (6044): No heartbeat from core client for 30 sec - exiting 14:41:35 (6044): No heartbeat from core client for 30 sec - exiting 14:41:36 (6044): No heartbeat from core client for 30 sec - exiting 14:41:37 (6044): No heartbeat from core client for 30 sec - exiting 14:41:38 (6044): No heartbeat from core client for 30 sec - exiting 14:41:39 (6044): No heartbeat from core client for 30 sec - exiting 14:41:40 (6044): No heartbeat from core client for 30 sec - exiting 14:41:41 (6044): No heartbeat from core client for 30 sec - exiting 14:41:42 (6044): No heartbeat from core client for 30 sec - exiting 14:41:43 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:33:55 (4356): No heartbeat from core client for 30 sec - exiting 08:33:57 (4356): No heartbeat from core client for 30 sec - exiting 08:33:58 (4356): No heartbeat from core client for 30 sec - exiting 08:33:59 (4356): No heartbeat from core client for 30 sec - exiting 08:34:00 (4356): No heartbeat from core client for 30 sec - exiting 08:34:01 (4356): No heartbeat from core client for 30 sec - exiting 08:34:02 (4356): No heartbeat from core client for 30 sec - exiting 08:34:03 (4356): No heartbeat from core client for 30 sec - exiting 08:34:04 (4356): No heartbeat from core client for 30 sec - exiting 08:34:05 (4356): No heartbeat from core client for 30 sec - exiting 08:34:06 (4356): No heartbeat from core client for 30 sec - exiting 08:34:08 (4356): No heartbeat from core client for 30 sec - exiting 08:34:09 (4356): No heartbeat from core client for 30 sec - exiting 08:34:10 (4356): No heartbeat from core client for 30 sec - exiting 08:34:11 (4356): No heartbeat from core client for 30 sec - exiting 08:34:12 (4356): No heartbeat from core client for 30 sec - exiting 08:34:13 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:14 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:23:05 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 10:51:17 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... 06:51:33 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:54 (1020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:22:35 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:42:40 (13324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... 18:54:32 (4564): No heartbeat from core client for 30 sec - exiting 18:54:33 (4564): No heartbeat from core client for 30 sec - exiting 18:54:34 (4564): No heartbeat from core client for 30 sec - exiting 18:54:35 (4564): No heartbeat from core client for 30 sec - exiting 18:54:36 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:05:16 (4824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:53 (752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:27 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... 18:26:14 (716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:56 (3628): No heartbeat from core client for 30 sec - exiting 22:46:57 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:13 (4316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:00 (4908): No heartbeat from core client for 30 sec - exiting 17:58:01 (4908): No heartbeat from core client for 30 sec - exiting 17:58:02 (4908): No heartbeat from core client for 30 sec - exiting 17:58:03 (4908): No heartbeat from core client for 30 sec - exiting 17:58:04 (4908): No heartbeat from core client for 30 sec - exiting 17:58:05 (4908): No heartbeat from core client for 30 sec - exiting 17:58:06 (4908): No heartbeat from core client for 30 sec - exiting 17:58:07 (4908): No heartbeat from core client for 30 sec - exiting 17:58:08 (4908): No heartbeat from core client for 30 sec - exiting 17:58:09 (4908): No heartbeat from core client for 30 sec - exiting 17:58:10 (4908): No heartbeat from core client for 30 sec - exiting 17:58:11 (4908): No heartbeat from core client for 30 sec - exiting 17:58:12 (4908): No heartbeat from core client for 30 sec - exiting 17:58:13 (4908): No heartbeat from core client for 30 sec - exiting 17:58:14 (4908): No heartbeat from core client for 30 sec - exiting 17:58:15 (4908): No heartbeat from core client for 30 sec - exiting 17:58:16 (4908): No heartbeat from core client for 30 sec - exiting 17:58:17 (4908): No heartbeat from core client for 30 sec - exiting 17:58:18 (4908): No heartbeat from core client for 30 sec - exiting 17:58:20 (4908): No heartbeat from core client for 30 sec - exiting 17:58:21 (4908): No heartbeat from core client for 30 sec - exiting 17:58:22 (4908): No heartbeat from core client for 30 sec - exiting 17:58:23 (4908): No heartbeat from core client for 30 sec - exiting 17:58:24 (4908): No heartbeat from core client for 30 sec - exiting 17:58:25 (4908): No heartbeat from core client for 30 sec - exiting 17:58:26 (4908): No heartbeat from core client for 30 sec - exiting 17:58:27 (4908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:47:54 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:03 (26532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:33 (3488): No heartbeat from core client for 30 sec - exiting 19:36:34 (3488): No heartbeat from core client for 30 sec - exiting 19:36:35 (3488): No heartbeat from core client for 30 sec - exiting 19:36:36 (3488): No heartbeat from core client for 30 sec - exiting 19:36:37 (3488): No heartbeat from core client for 30 sec - exiting 19:36:38 (3488): No heartbeat from core client for 30 sec - exiting 19:36:39 (3488): No heartbeat from core client for 30 sec - exiting 19:36:40 (3488): No heartbeat from core client for 30 sec - exiting 19:36:42 (3488): No heartbeat from core client for 30 sec - exiting 19:36:43 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:37:46 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:14 (1444): No heartbeat from core client for 30 sec - exiting 17:23:15 (1444): No heartbeat from core client for 30 sec - exiting 17:23:16 (1444): No heartbeat from core client for 30 sec - exiting 17:23:17 (1444): No heartbeat from core client for 30 sec - exiting 17:23:18 (1444): No heartbeat from core client for 30 sec - exiting 17:23:19 (1444): No heartbeat from core client for 30 sec - exiting 17:23:20 (1444): No heartbeat from core client for 30 sec - exiting 17:23:21 (1444): No heartbeat from core client for 30 sec - exiting 17:23:22 (1444): No heartbeat from core client for 30 sec - exiting 17:23:23 (1444): No heartbeat from core client for 30 sec - exiting 17:23:24 (1444): No heartbeat from core client for 30 sec - exiting 17:23:25 (1444): No heartbeat from core client for 30 sec - exiting 17:23:26 (1444): No heartbeat from core client for 30 sec - exiting 17:23:27 (1444): No heartbeat from core client for 30 sec - exiting 17:23:28 (1444): No heartbeat from core client for 30 sec - exiting 17:23:29 (1444): No heartbeat from core client for 30 sec - exiting 17:23:30 (1444): No heartbeat from core client for 30 sec - exiting 17:23:31 (1444): No heartbeat from core client for 30 sec - exiting 17:23:32 (1444): No heartbeat from core client for 30 sec - exiting 17:23:33 (1444): No heartbeat from core client for 30 sec - exiting 17:23:34 (1444): No heartbeat from core client for 30 sec - exiting 17:23:35 (1444): No heartbeat from core client for 30 sec - exiting 17:23:36 (1444): No heartbeat from core client for 30 sec - exiting 17:23:37 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:52 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:54 (3932): No heartbeat from core client for 30 sec - exiting 17:32:55 (3932): No heartbeat from core client for 30 sec - exiting 17:34:11 (780): No heartbeat from core client for 30 sec - exiting 17:34:12 (780): No heartbeat from core client for 30 sec - exiting 17:34:13 (780): No heartbeat from core client for 30 sec - exiting 17:34:14 (780): No heartbeat from core client for 30 sec - exiting 17:34:16 (780): No heartbeat from core client for 30 sec - exiting 17:34:17 (780): No heartbeat from core client for 30 sec - exiting 17:34:18 (780): No heartbeat from core client for 30 sec - exiting 17:34:19 (780): No heartbeat from core client for 30 sec - exiting 17:34:20 (780): No heartbeat from core client for 30 sec - exiting 17:34:21 (780): No heartbeat from core client for 30 sec - exiting 17:34:22 (780): No heartbeat from core client for 30 sec - exiting 17:34:23 (780): No heartbeat from core client for 30 sec - exiting 17:34:24 (780): No heartbeat from core client for 30 sec - exiting 17:34:25 (780): No heartbeat from core client for 30 sec - exiting 17:34:26 (780): No heartbeat from core client for 30 sec - exiting 17:34:28 (780): No heartbeat from core client for 30 sec - exiting 17:34:29 (780): No heartbeat from core client for 30 sec - exiting 17:34:30 (780): No heartbeat from core client for 30 sec - exiting 17:34:31 (780): No heartbeat from core client for 30 sec - exiting 17:34:32 (780): No heartbeat from core client for 30 sec - exiting 17:34:33 (780): No heartbeat from core client for 30 sec - exiting 17:34:34 (780): No heartbeat from core client for 30 sec - exiting 17:34:35 (780): No heartbeat from core client for 30 sec - exiting 17:34:36 (780): No heartbeat from core client for 30 sec - exiting 17:34:37 (780): No heartbeat from core client for 30 sec - exiting 17:34:38 (780): No heartbeat from core client for 30 sec - exiting 17:34:40 (780): No heartbeat from core client for 30 sec - exiting 17:34:41 (780): No heartbeat from core client for 30 sec - exiting 17:34:42 (780): No heartbeat from core client for 30 sec - exiting 17:34:43 (780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:14 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:05 (40508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:07 (44144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:11 (43472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:14 (43028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:22 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:25 (45708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:13 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... 09:33:47 (4236): No heartbeat from core client for 30 sec - exiting 09:33:48 (4236): No heartbeat from core client for 30 sec - exiting 09:33:49 (4236): No heartbeat from core client for 30 sec - exiting 09:33:50 (4236): No heartbeat from core client for 30 sec - exiting 09:33:51 (4236): No heartbeat from core client for 30 sec - exiting 09:33:52 (4236): No heartbeat from core client for 30 sec - exiting 09:33:53 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:49 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:41 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:42 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:21:46 (36632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:02 (36784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:07 (39480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jun 2012 21:07:11	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	311,040	422,488	1.3583
23 Jun 2012 09:29:05	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	285,120	387,138	1.3578
22 Jun 2012 04:05:07	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	259,200	351,906	1.3577
21 Jun 2012 06:03:42	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	233,280	317,012	1.3589
19 Jun 2012 17:58:26	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	207,360	281,011	1.3552
12 Jun 2012 17:30:08	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	181,440	246,000	1.3558
10 Jun 2012 09:18:28	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	155,520	211,214	1.3581
08 Jun 2012 15:43:26	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	129,600	175,673	1.3555
05 Jun 2012 19:27:53	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	103,680	141,675	1.3665
04 Jun 2012 13:41:32	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	77,760	106,023	1.3635
02 Jun 2012 12:28:28	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	51,840	70,422	1.3584
01 Jun 2012 16:08:01	1036046	14761178	hadcm3n_o1uk_2020_40_007999003_0	25,920	35,164	1.3566