Task 15470273

Name	hadcm3n_zf5t_1920_40_008255007_1
Workunit	8410131
Created	4 Dec 2012, 0:33:41 UTC
Sent	4 Dec 2012, 0:33:59 UTC
Report deadline	5 Mar 2013, 8:01:10 UTC
Received	29 Jan 2013, 19:39:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1147281
Run time	11 days 3 hours 24 min 43 sec
CPU time	10 days 8 hours 17 min 12 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:58:38 (4964): No heartbeat from core client for 30 sec - exiting 19:58:39 (4964): No heartbeat from core client for 30 sec - exiting 19:58:40 (4964): No heartbeat from core client for 30 sec - exiting 19:58:41 (4964): No heartbeat from core client for 30 sec - exiting 19:58:42 (4964): No heartbeat from core client for 30 sec - exiting 19:58:43 (4964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:27 (5104): No heartbeat from core client for 30 sec - exiting 22:38:28 (5104): No heartbeat from core client for 30 sec - exiting 22:38:29 (5104): No heartbeat from core client for 30 sec - exiting 22:38:30 (5104): No heartbeat from core client for 30 sec - exiting 22:38:31 (5104): No heartbeat from core client for 30 sec - exiting 22:38:32 (5104): No heartbeat from core client for 30 sec - exiting 22:38:34 (5104): No heartbeat from core client for 30 sec - exiting 22:38:35 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:36 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:41:19 (1664): No heartbeat from core client for 30 sec - exiting 20:41:20 (1664): No heartbeat from core client for 30 sec - exiting 20:41:21 (1664): No heartbeat from core client for 30 sec - exiting 20:41:22 (1664): No heartbeat from core client for 30 sec - exiting 20:41:23 (1664): No heartbeat from core client for 30 sec - exiting 20:41:25 (1664): No heartbeat from core client for 30 sec - exiting 20:41:26 (1664): No heartbeat from core client for 30 sec - exiting 20:41:27 (1664): No heartbeat from core client for 30 sec - exiting 20:41:28 (1664): No heartbeat from core client for 30 sec - exiting 20:41:29 (1664): No heartbeat from core client for 30 sec - exiting 20:41:30 (1664): No heartbeat from core client for 30 sec - exiting 20:41:31 (1664): No heartbeat from core client for 30 sec - exiting 20:41:32 (1664): No heartbeat from core client for 30 sec - exiting 20:41:33 (1664): No heartbeat from core client for 30 sec - exiting 20:41:34 (1664): No heartbeat from core client for 30 sec - exiting 20:41:35 (1664): No heartbeat from core client for 30 sec - exiting 20:41:36 (1664): No heartbeat from core client for 30 sec - exiting 20:41:38 (1664): No heartbeat from core client for 30 sec - exiting 20:41:39 (1664): No heartbeat from core client for 30 sec - exiting 20:41:40 (1664): No heartbeat from core client for 30 sec - exiting 20:41:41 (1664): No heartbeat from core client for 30 sec - exiting 20:41:42 (1664): No heartbeat from core client for 30 sec - exiting 20:41:43 (1664): No heartbeat from core client for 30 sec - exiting 20:41:44 (1664): No heartbeat from core client for 30 sec - exiting 20:41:45 (1664): No heartbeat from core client for 30 sec - exiting 20:41:46 (1664): No heartbeat from core client for 30 sec - exiting 20:41:47 (1664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:49 (1664): No heartbeat from core client for 30 sec - exiting 20:41:50 (1664): No heartbeat from core client for 30 sec - exiting 20:41:51 (1664): No heartbeat from core client for 30 sec - exiting 20:41:52 (1664): No heartbeat from core client for 30 sec - exiting 20:41:53 (1664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:08:49 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:50:24 (3304): No heartbeat from core client for 30 sec - exiting 22:50:25 (3304): No heartbeat from core client for 30 sec - exiting 22:50:27 (3304): No heartbeat from core client for 30 sec - exiting 22:50:28 (3304): No heartbeat from core client for 30 sec - exiting 22:50:29 (3304): No heartbeat from core client for 30 sec - exiting 22:50:30 (3304): No heartbeat from core client for 30 sec - exiting 22:50:31 (3304): No heartbeat from core client for 30 sec - exiting 22:50:32 (3304): No heartbeat from core client for 30 sec - exiting 22:50:33 (3304): No heartbeat from core client for 30 sec - exiting 22:50:34 (3304): No heartbeat from core client for 30 sec - exiting 22:50:35 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:50:36 (3304): No heartbeat from core client for 30 sec - exiting 22:50:37 (3304): No heartbeat from core client for 30 sec - exiting 23:15:49 (5184): No heartbeat from core client for 30 sec - exiting 23:15:51 (5184): No heartbeat from core client for 30 sec - exiting 23:15:52 (5184): No heartbeat from core client for 30 sec - exiting 23:15:53 (5184): No heartbeat from core client for 30 sec - exiting 23:15:54 (5184): No heartbeat from core client for 30 sec - exiting 23:15:55 (5184): No heartbeat from core client for 30 sec - exiting 23:15:56 (5184): No heartbeat from core client for 30 sec - exiting 23:15:57 (5184): No heartbeat from core client for 30 sec - exiting 23:15:58 (5184): No heartbeat from core client for 30 sec - exiting 23:15:59 (5184): No heartbeat from core client for 30 sec - exiting 23:16:00 (5184): No heartbeat from core client for 30 sec - exiting 23:16:01 (5184): No heartbeat from core client for 30 sec - exiting 23:16:03 (5184): No heartbeat from core client for 30 sec - exiting 23:16:04 (5184): No heartbeat from core client for 30 sec - exiting 23:16:05 (5184): No heartbeat from core client for 30 sec - exiting 23:16:06 (5184): No heartbeat from core client for 30 sec - exiting 23:16:07 (5184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:24 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:35 (3512): No heartbeat from core client for 30 sec - exiting 11:57:36 (3512): No heartbeat from core client for 30 sec - exiting 11:57:37 (3512): No heartbeat from core client for 30 sec - exiting 11:57:39 (3512): No heartbeat from core client for 30 sec - exiting 11:57:40 (3512): No heartbeat from core client for 30 sec - exiting 11:57:41 (3512): No heartbeat from core client for 30 sec - exiting 11:57:42 (3512): No heartbeat from core client for 30 sec - exiting 11:57:43 (3512): No heartbeat from core client for 30 sec - exiting 11:57:44 (3512): No heartbeat from core client for 30 sec - exiting 11:57:45 (3512): No heartbeat from core client for 30 sec - exiting 11:57:46 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:57 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:33:23 (1808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:44:24 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:20 (3928): No heartbeat from core client for 30 sec - exiting 12:05:21 (3928): No heartbeat from core client for 30 sec - exiting 12:05:22 (3928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:38:10 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:51 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5520, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4236, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5544, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:36:06 (4068): No heartbeat from core client for 30 sec - exiting 08:36:07 (4068): No heartbeat from core client for 30 sec - exiting 08:36:08 (4068): No heartbeat from core client for 30 sec - exiting 08:36:09 (4068): No heartbeat from core client for 30 sec - exiting 08:36:10 (4068): No heartbeat from core client for 30 sec - exiting 08:36:11 (4068): No heartbeat from core client for 30 sec - exiting 08:36:13 (4068): No heartbeat from core client for 30 sec - exiting 08:36:14 (4068): No heartbeat from core client for 30 sec - exiting 08:36:15 (4068): No heartbeat from core client for 30 sec - exiting 08:36:16 (4068): No heartbeat from core client for 30 sec - exiting 08:36:17 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:18 (4068): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jan 2013 19:41:17	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	518,400	893,830	1.7242
28 Jan 2013 09:34:38	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	492,480	849,839	1.7256
27 Jan 2013 14:02:06	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	466,560	806,401	1.7284
26 Jan 2013 13:45:19	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	440,640	763,698	1.7332
23 Jan 2013 21:53:30	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	414,720	718,423	1.7323
21 Jan 2013 18:51:48	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	388,800	672,132	1.7287
17 Jan 2013 20:36:37	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	362,880	629,726	1.7354
13 Jan 2013 21:49:32	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	336,960	586,474	1.7405
12 Jan 2013 13:25:13	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	311,040	539,961	1.7360
31 Dec 2012 13:40:31	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	285,120	496,701	1.7421
30 Dec 2012 13:50:32	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	259,200	449,702	1.7350
22 Dec 2012 19:06:59	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	233,280	404,037	1.7320
19 Dec 2012 12:57:44	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	207,360	357,761	1.7253
17 Dec 2012 21:49:35	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	181,440	312,692	1.7234
15 Dec 2012 17:31:21	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	155,520	267,927	1.7228
14 Dec 2012 10:06:15	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	129,600	222,009	1.7130
13 Dec 2012 22:07:22	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	103,680	178,040	1.7172
13 Dec 2012 22:07:22	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	77,760	133,349	1.7149
13 Dec 2012 22:07:22	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	51,840	89,218	1.7210
05 Dec 2012 22:21:14	1147281	15470273	hadcm3n_zf5t_1920_40_008255007_1	25,920	45,068	1.7387