Task 13824108

Name	hadcm3n_ya20_1940_40_007617068_4
Workunit	7795198
Created	27 Dec 2011, 18:09:42 UTC
Sent	27 Dec 2011, 18:09:47 UTC
Report deadline	28 Mar 2012, 1:36:58 UTC
Received	18 Jan 2012, 0:37:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1074891
Run time	20 days 10 hours 12 min 13 sec
CPU time	19 days 21 hours 16 min 7 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 23:13:18 (2156): No heartbeat from core client for 30 sec - exiting 23:13:19 (2156): No heartbeat from core client for 30 sec - exiting 23:13:20 (2156): No heartbeat from core client for 30 sec - exiting 23:13:21 (2156): No heartbeat from core client for 30 sec - exiting 23:13:22 (2156): No heartbeat from core client for 30 sec - exiting 23:13:23 (2156): No heartbeat from core client for 30 sec - exiting 23:13:24 (2156): No heartbeat from core client for 30 sec - exiting 23:13:25 (2156): No heartbeat from core client for 30 sec - exiting 23:13:26 (2156): No heartbeat from core client for 30 sec - exiting 23:13:27 (2156): No heartbeat from core client for 30 sec - exiting 23:13:28 (2156): No heartbeat from core client for 30 sec - exiting 23:13:29 (2156): No heartbeat from core client for 30 sec - exiting 23:13:30 (2156): No heartbeat from core client for 30 sec - exiting 23:13:31 (2156): No heartbeat from core client for 30 sec - exiting 23:13:32 (2156): No heartbeat from core client for 30 sec - exiting 23:13:33 (2156): No heartbeat from core client for 30 sec - exiting 23:13:34 (2156): No heartbeat from core client for 30 sec - exiting 23:13:35 (2156): No heartbeat from core client for 30 sec - exiting 23:13:36 (2156): No heartbeat from core client for 30 sec - exiting 23:13:37 (2156): No heartbeat from core client for 30 sec - exiting 23:13:38 (2156): No heartbeat from core client for 30 sec - exiting 23:13:39 (2156): No heartbeat from core client for 30 sec - exiting 23:13:40 (2156): No heartbeat from core client for 30 sec - exiting 23:13:41 (2156): No heartbeat from core client for 30 sec - exiting 23:13:42 (2156): No heartbeat from core client for 30 sec - exiting 23:13:43 (2156): No heartbeat from core client for 30 sec - exiting 23:13:44 (2156): No heartbeat from core client for 30 sec - exiting 23:13:45 (2156): No heartbeat from core client for 30 sec - exiting 23:13:46 (2156): No heartbeat from core client for 30 sec - exiting 23:13:47 (2156): No heartbeat from core client for 30 sec - exiting 23:13:48 (2156): No heartbeat from core client for 30 sec - exiting 23:13:49 (2156): No heartbeat from core client for 30 sec - exiting 23:13:50 (2156): No heartbeat from core client for 30 sec - exiting 23:13:51 (2156): No heartbeat from core client for 30 sec - exiting 23:13:52 (2156): No heartbeat from core client for 30 sec - exiting 23:13:53 (2156): No heartbeat from core client for 30 sec - exiting 23:13:54 (2156): No heartbeat from core client for 30 sec - exiting 23:13:55 (2156): No heartbeat from core client for 30 sec - exiting 23:13:56 (2156): No heartbeat from core client for 30 sec - exiting 23:13:57 (2156): No heartbeat from core client for 30 sec - exiting 23:13:58 (2156): No heartbeat from core client for 30 sec - exiting 23:13:59 (2156): No heartbeat from core client for 30 sec - exiting 23:14:00 (2156): No heartbeat from core client for 30 sec - exiting 23:14:01 (2156): No heartbeat from core client for 30 sec - exiting 23:14:02 (2156): No heartbeat from core client for 30 sec - exiting 23:14:03 (2156): No heartbeat from core client for 30 sec - exiting 23:14:04 (2156): No heartbeat from core client for 30 sec - exiting 23:14:05 (2156): No heartbeat from core client for 30 sec - exiting 23:14:06 (2156): No heartbeat from core client for 30 sec - exiting 23:14:07 (2156): No heartbeat from core client for 30 sec - exiting 23:14:08 (2156): No heartbeat from core client for 30 sec - exiting 23:14:09 (2156): No heartbeat from core client for 30 sec - exiting 23:14:10 (2156): No heartbeat from core client for 30 sec - exiting 23:14:11 (2156): No heartbeat from core client for 30 sec - exiting 23:14:12 (2156): No heartbeat from core client for 30 sec - exiting 23:14:13 (2156): No heartbeat from core client for 30 sec - exiting 23:14:14 (2156): No heartbeat from core client for 30 sec - exiting 23:14:15 (2156): No heartbeat from core client for 30 sec - exiting 23:14:16 (2156): No heartbeat from core client for 30 sec - exiting 23:14:17 (2156): No heartbeat from core client for 30 sec - exiting 23:14:18 (2156): No heartbeat from core client for 30 sec - exiting 23:14:19 (2156): No heartbeat from core client for 30 sec - exiting 23:14:20 (2156): No heartbeat from core client for 30 sec - exiting 23:14:21 (2156): No heartbeat from core client for 30 sec - exiting 23:14:22 (2156): No heartbeat from core client for 30 sec - exiting 23:14:23 (2156): No heartbeat from core client for 30 sec - exiting 23:14:24 (2156): No heartbeat from core client for 30 sec - exiting 23:14:25 (2156): No heartbeat from core client for 30 sec - exiting 23:14:26 (2156): No heartbeat from core client for 30 sec - exiting 23:14:27 (2156): No heartbeat from core client for 30 sec - exiting 23:14:28 (2156): No heartbeat from core client for 30 sec - exiting 23:14:29 (2156): No heartbeat from core client for 30 sec - exiting 23:14:30 (2156): No heartbeat from core client for 30 sec - exiting 23:14:31 (2156): No heartbeat from core client for 30 sec - exiting 23:14:32 (2156): No heartbeat from core client for 30 sec - exiting 23:14:33 (2156): No heartbeat from core client for 30 sec - exiting 23:14:34 (2156): No heartbeat from core client for 30 sec - exiting 23:14:35 (2156): No heartbeat from core client for 30 sec - exiting 23:14:36 (2156): No heartbeat from core client for 30 sec - exiting 23:14:37 (2156): No heartbeat from core client for 30 sec - exiting 23:14:38 (2156): No heartbeat from core client for 30 sec - exiting 23:14:39 (2156): No heartbeat from core client for 30 sec - exiting 23:14:40 (2156): No heartbeat from core client for 30 sec - exiting 23:14:41 (2156): No heartbeat from core client for 30 sec - exiting 23:14:42 (2156): No heartbeat from core client for 30 sec - exiting 23:14:43 (2156): No heartbeat from core client for 30 sec - exiting 23:14:44 (2156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... 23:27:07 (5680): No heartbeat from core client for 30 sec - exiting 23:27:08 (5680): No heartbeat from core client for 30 sec - exiting 23:27:09 (5680): No heartbeat from core client for 30 sec - exiting 23:27:10 (5680): No heartbeat from core client for 30 sec - exiting 23:27:11 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:04:48 (7148): No heartbeat from core client for 30 sec - exiting 17:04:49 (7148): No heartbeat from core client for 30 sec - exiting 17:04:50 (7148): No heartbeat from core client for 30 sec - exiting 17:04:51 (7148): No heartbeat from core client for 30 sec - exiting 17:04:52 (7148): No heartbeat from core client for 30 sec - exiting 17:04:53 (7148): No heartbeat from core client for 30 sec - exiting 17:04:54 (7148): No heartbeat from core client for 30 sec - exiting 17:04:57 (7148): No heartbeat from core client for 30 sec - exiting 17:04:58 (7148): No heartbeat from core client for 30 sec - exiting 17:04:59 (7148): No heartbeat from core client for 30 sec - exiting 17:05:01 (7148): No heartbeat from core client for 30 sec - exiting 17:05:02 (7148): No heartbeat from core client for 30 sec - exiting 17:05:03 (7148): No heartbeat from core client for 30 sec - exiting 17:05:04 (7148): No heartbeat from core client for 30 sec - exiting 17:05:05 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:29:44 (3556): No heartbeat from core client for 30 sec - exiting 15:29:45 (3556): No heartbeat from core client for 30 sec - exiting 15:29:46 (3556): No heartbeat from core client for 30 sec - exiting 15:29:47 (3556): No heartbeat from core client for 30 sec - exiting 15:29:48 (3556): No heartbeat from core client for 30 sec - exiting 15:29:49 (3556): No heartbeat from core client for 30 sec - exiting 15:29:50 (3556): No heartbeat from core client for 30 sec - exiting 15:29:51 (3556): No heartbeat from core client for 30 sec - exiting 15:29:52 (3556): No heartbeat from core client for 30 sec - exiting 15:29:53 (3556): No heartbeat from core client for 30 sec - exiting 15:29:54 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774D3232 read attempt to address 0x40A67181 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ya20_1940_40_007617068/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jan 2012 00:42:15	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	777,600	1,718,164	2.2096
17 Jan 2012 08:03:00	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	751,680	1,658,611	2.2065
16 Jan 2012 13:36:17	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	725,760	1,600,090	2.2047
15 Jan 2012 22:22:18	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	699,840	1,542,782	2.2045
15 Jan 2012 06:29:57	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	673,920	1,489,253	2.2098
14 Jan 2012 13:03:04	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	648,000	1,433,325	2.2119
13 Jan 2012 19:11:26	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	622,080	1,377,679	2.2146
12 Jan 2012 23:45:42	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	596,160	1,321,641	2.2169
12 Jan 2012 07:02:45	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	570,240	1,265,951	2.2200
11 Jan 2012 15:05:41	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	544,320	1,209,481	2.2220
10 Jan 2012 22:51:31	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	518,400	1,152,886	2.2239
10 Jan 2012 06:07:45	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	492,480	1,096,764	2.2270
09 Jan 2012 14:29:47	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	466,560	1,040,173	2.2295
09 Jan 2012 01:56:40	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	440,640	983,813	2.2327
08 Jan 2012 10:55:21	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	414,720	926,723	2.2346
07 Jan 2012 14:33:26	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	388,800	869,563	2.2365
06 Jan 2012 23:15:08	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	362,880	812,397	2.2387
06 Jan 2012 05:22:20	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	336,960	755,263	2.2414
05 Jan 2012 14:30:56	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	311,040	697,241	2.2416
04 Jan 2012 20:10:27	1074891	13824108	hadcm3n_ya20_1940_40_007617068_4	285,120	637,937	2.2374