Task 15068204

Name	hadcm3n_o650_1980_40_008116599_1
Workunit	8271713
Created	3 Aug 2012, 19:51:37 UTC
Sent	3 Aug 2012, 19:51:49 UTC
Report deadline	3 Nov 2012, 3:19:00 UTC
Received	17 Sep 2012, 23:49:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	927193
Run time	22 days 18 hours 52 min 24 sec
CPU time	21 days 1 hours 18 min 37 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7004, iMonCtr=1 Model crash detected, will try to restart... 22:16:16 (4712): No heartbeat from core client for 30 sec - exiting 22:16:17 (4712): No heartbeat from core client for 30 sec - exiting 22:16:18 (4712): No heartbeat from core client for 30 sec - exiting 22:16:19 (4712): No heartbeat from core client for 30 sec - exiting 22:16:20 (4712): No heartbeat from core client for 30 sec - exiting 22:16:22 (4712): No heartbeat from core client for 30 sec - exiting 22:16:23 (4712): No heartbeat from core client for 30 sec - exiting 22:16:24 (4712): No heartbeat from core client for 30 sec - exiting 22:16:25 (4712): No heartbeat from core client for 30 sec - exiting 22:16:26 (4712): No heartbeat from core client for 30 sec - exiting 22:16:27 (4712): No heartbeat from core client for 30 sec - exiting 22:16:28 (4712): No heartbeat from core client for 30 sec - exiting 22:16:29 (4712): No heartbeat from core client for 30 sec - exiting 22:16:30 (4712): No heartbeat from core client for 30 sec - exiting 22:16:31 (4712): No heartbeat from core client for 30 sec - exiting 22:16:33 (4712): No heartbeat from core client for 30 sec - exiting 22:16:34 (4712): No heartbeat from core client for 30 sec - exiting 22:16:35 (4712): No heartbeat from core client for 30 sec - exiting 22:16:36 (4712): No heartbeat from core client for 30 sec - exiting 22:16:37 (4712): No heartbeat from core client for 30 sec - exiting 22:16:38 (4712): No heartbeat from core client for 30 sec - exiting 22:16:39 (4712): No heartbeat from core client for 30 sec - exiting 22:16:40 (4712): No heartbeat from core client for 30 sec - exiting 22:16:41 (4712): No heartbeat from core client for 30 sec - exiting 22:16:42 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:02:09 (5988): No heartbeat from core client for 30 sec - exiting 00:02:10 (5988): No heartbeat from core client for 30 sec - exiting 00:02:11 (5988): No heartbeat from core client for 30 sec - exiting 00:02:12 (5988): No heartbeat from core client for 30 sec - exiting 00:02:14 (5988): No heartbeat from core client for 30 sec - exiting 00:02:15 (5988): No heartbeat from core client for 30 sec - exiting 00:02:16 (5988): No heartbeat from core client for 30 sec - exiting 00:02:17 (5988): No heartbeat from core client for 30 sec - exiting 00:02:18 (5988): No heartbeat from core client for 30 sec - exiting 00:02:19 (5988): No heartbeat from core client for 30 sec - exiting 00:02:20 (5988): No heartbeat from core client for 30 sec - exiting 00:02:21 (5988): No heartbeat from core client for 30 sec - exiting 00:02:22 (5988): No heartbeat from core client for 30 sec - exiting 00:02:23 (5988): No heartbeat from core client for 30 sec - exiting 00:02:24 (5988): No heartbeat from core client for 30 sec - exiting 00:02:26 (5988): No heartbeat from core client for 30 sec - exiting 00:02:27 (5988): No heartbeat from core client for 30 sec - exiting 00:02:28 (5988): No heartbeat from core client for 30 sec - exiting 00:02:29 (5988): No heartbeat from core client for 30 sec - exiting 00:02:30 (5988): No heartbeat from core client for 30 sec - exiting 00:02:31 (5988): No heartbeat from core client for 30 sec - exiting 00:02:32 (5988): No heartbeat from core client for 30 sec - exiting 00:02:33 (5988): No heartbeat from core client for 30 sec - exiting 00:02:34 (5988): No heartbeat from core client for 30 sec - exiting 00:02:35 (5988): No heartbeat from core client for 30 sec - exiting 00:02:37 (5988): No heartbeat from core client for 30 sec - exiting 00:02:38 (5988): No heartbeat from core client for 30 sec - exiting 00:02:39 (5988): No heartbeat from core client for 30 sec - exiting 00:02:40 (5988): No heartbeat from core client for 30 sec - exiting 00:02:41 (5988): No heartbeat from core client for 30 sec - exiting 00:02:42 (5988): No heartbeat from core client for 30 sec - exiting 00:02:43 (5988): No heartbeat from core client for 30 sec - exiting 00:02:44 (5988): No heartbeat from core client for 30 sec - exiting 00:02:45 (5988): No heartbeat from core client for 30 sec - exiting 00:02:46 (5988): No heartbeat from core client for 30 sec - exiting 00:02:47 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:24:56 (7248): No heartbeat from core client for 30 sec - exiting 19:24:57 (7248): No heartbeat from core client for 30 sec - exiting 19:24:58 (7248): No heartbeat from core client for 30 sec - exiting 19:25:00 (7248): No heartbeat from core client for 30 sec - exiting 19:25:01 (7248): No heartbeat from core client for 30 sec - exiting 19:25:02 (7248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:32:50 (11584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:51 (11584): No heartbeat from core client for 30 sec - exiting 00:46:32 (8144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:46:33 (8144): No heartbeat from core client for 30 sec - exiting 04:42:38 (10276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:20 (8544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:01 (11224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:25:31 (7560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 21:40:58 (11560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:55:08 (12756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:09:33 (4672): No heartbeat from core client for 30 sec - exiting 13:09:34 (4672): No heartbeat from core client for 30 sec - exiting 13:09:35 (4672): No heartbeat from core client for 30 sec - exiting 13:09:36 (4672): No heartbeat from core client for 30 sec - exiting 13:09:37 (4672): No heartbeat from core client for 30 sec - exiting 13:09:38 (4672): No heartbeat from core client for 30 sec - exiting 13:09:39 (4672): No heartbeat from core client for 30 sec - exiting 13:09:40 (4672): No heartbeat from core client for 30 sec - exiting 13:09:41 (4672): No heartbeat from core client for 30 sec - exiting 13:09:42 (4672): No heartbeat from core client for 30 sec - exiting 13:09:44 (4672): No heartbeat from core client for 30 sec - exiting 13:09:45 (4672): No heartbeat from core client for 30 sec - exiting 13:09:46 (4672): No heartbeat from core client for 30 sec - exiting 13:09:47 (4672): No heartbeat from core client for 30 sec - exiting 13:09:48 (4672): No heartbeat from core client for 30 sec - exiting 13:09:49 (4672): No heartbeat from core client for 30 sec - exiting 13:09:50 (4672): No heartbeat from core client for 30 sec - exiting 13:09:51 (4672): No heartbeat from core client for 30 sec - exiting 13:09:52 (4672): No heartbeat from core client for 30 sec - exiting 13:09:53 (4672): No heartbeat from core client for 30 sec - exiting 13:09:54 (4672): No heartbeat from core client for 30 sec - exiting 13:09:56 (4672): No heartbeat from core client for 30 sec - exiting 13:09:57 (4672): No heartbeat from core client for 30 sec - exiting 13:09:58 (4672): No heartbeat from core client for 30 sec - exiting 13:09:59 (4672): No heartbeat from core client for 30 sec - exiting 13:10:00 (4672): No heartbeat from core client for 30 sec - exiting 13:10:01 (4672): No heartbeat from core client for 30 sec - exiting 13:10:02 (4672): No heartbeat from core client for 30 sec - exiting 13:10:03 (4672): No heartbeat from core client for 30 sec - exiting 13:10:04 (4672): No heartbeat from core client for 30 sec - exiting 13:10:05 (4672): No heartbeat from core client for 30 sec - exiting 13:10:07 (4672): No heartbeat from core client for 30 sec - exiting 13:10:08 (4672): No heartbeat from core client for 30 sec - exiting 13:10:09 (4672): No heartbeat from core client for 30 sec - exiting 13:10:10 (4672): No heartbeat from core client for 30 sec - exiting 13:10:11 (4672): No heartbeat from core client for 30 sec - exiting 13:10:12 (4672): No heartbeat from core client for 30 sec - exiting 13:10:13 (4672): No heartbeat from core client for 30 sec - exiting 13:10:14 (4672): No heartbeat from core client for 30 sec - exiting 13:10:15 (4672): No heartbeat from core client for 30 sec - exiting 13:10:17 (4672): No heartbeat from core client for 30 sec - exiting 13:10:18 (4672): No heartbeat from core client for 30 sec - exiting 13:10:19 (4672): No heartbeat from core client for 30 sec - exiting 13:10:20 (4672): No heartbeat from core client for 30 sec - exiting 13:10:21 (4672): No heartbeat from core client for 30 sec - exiting 13:10:22 (4672): No heartbeat from core client for 30 sec - exiting 13:10:23 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:13:11 (11432): No heartbeat from core client for 30 sec - exiting 19:13:13 (11432): No heartbeat from core client for 30 sec - exiting 19:13:14 (11432): No heartbeat from core client for 30 sec - exiting 19:13:15 (11432): No heartbeat from core client for 30 sec - exiting 19:13:16 (11432): No heartbeat from core client for 30 sec - exiting 19:13:17 (11432): No heartbeat from core client for 30 sec - exiting 19:13:18 (11432): No heartbeat from core client for 30 sec - exiting 19:13:19 (11432): No heartbeat from core client for 30 sec - exiting 19:13:20 (11432): No heartbeat from core client for 30 sec - exiting 19:13:21 (11432): No heartbeat from core client for 30 sec - exiting 19:13:22 (11432): No heartbeat from core client for 30 sec - exiting 19:13:23 (11432): No heartbeat from core client for 30 sec - exiting 19:13:24 (11432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:23:09 (5448): No heartbeat from core client for 30 sec - exiting 09:23:10 (5448): No heartbeat from core client for 30 sec - exiting 09:23:11 (5448): No heartbeat from core client for 30 sec - exiting 09:23:12 (5448): No heartbeat from core client for 30 sec - exiting 09:23:13 (5448): No heartbeat from core client for 30 sec - exiting 09:23:14 (5448): No heartbeat from core client for 30 sec - exiting 09:23:16 (5448): No heartbeat from core client for 30 sec - exiting 09:23:17 (5448): No heartbeat from core client for 30 sec - exiting 09:23:18 (5448): No heartbeat from core client for 30 sec - exiting 09:23:19 (5448): No heartbeat from core client for 30 sec - exiting 09:23:20 (5448): No heartbeat from core client for 30 sec - exiting 09:23:21 (5448): No heartbeat from core client for 30 sec - exiting 09:23:22 (5448): No heartbeat from core client for 30 sec - exiting 09:23:23 (5448): No heartbeat from core client for 30 sec - exiting 09:23:24 (5448): No heartbeat from core client for 30 sec - exiting 09:23:25 (5448): No heartbeat from core client for 30 sec - exiting 09:23:27 (5448): No heartbeat from core client for 30 sec - exiting 09:23:28 (5448): No heartbeat from core client for 30 sec - exiting 09:23:29 (5448): No heartbeat from core client for 30 sec - exiting 09:23:30 (5448): No heartbeat from core client for 30 sec - exiting 09:23:31 (5448): No heartbeat from core client for 30 sec - exiting 09:23:32 (5448): No heartbeat from core client for 30 sec - exiting 09:23:33 (5448): No heartbeat from core client for 30 sec - exiting 09:23:34 (5448): No heartbeat from core client for 30 sec - exiting 09:23:35 (5448): No heartbeat from core client for 30 sec - exiting 09:23:36 (5448): No heartbeat from core client for 30 sec - exiting 09:23:37 (5448): No heartbeat from core client for 30 sec - exiting 09:23:39 (5448): No heartbeat from core client for 30 sec - exiting 09:23:40 (5448): No heartbeat from core client for 30 sec - exiting 09:23:41 (5448): No heartbeat from core client for 30 sec - exiting 09:23:42 (5448): No heartbeat from core client for 30 sec - exiting 09:23:43 (5448): No heartbeat from core client for 30 sec - exiting 09:23:44 (5448): No heartbeat from core client for 30 sec - exiting 09:23:45 (5448): No heartbeat from core client for 30 sec - exiting 09:23:46 (5448): No heartbeat from core client for 30 sec - exiting 09:23:47 (5448): No heartbeat from core client for 30 sec - exiting 09:23:48 (5448): No heartbeat from core client for 30 sec - exiting 09:23:50 (5448): No heartbeat from core client for 30 sec - exiting 09:23:51 (5448): No heartbeat from core client for 30 sec - exiting 09:23:52 (5448): No heartbeat from core client for 30 sec - exiting 09:23:53 (5448): No heartbeat from core client for 30 sec - exiting 09:23:54 (5448): No heartbeat from core client for 30 sec - exiting 09:23:55 (5448): No heartbeat from core client for 30 sec - exiting 09:23:56 (5448): No heartbeat from core client for 30 sec - exiting 09:23:57 (5448): No heartbeat from core client for 30 sec - exiting 09:23:58 (5448): No heartbeat from core client for 30 sec - exiting 09:23:59 (5448): No heartbeat from core client for 30 sec - exiting 09:24:01 (5448): No heartbeat from core client for 30 sec - exiting 09:24:02 (5448): No heartbeat from core client for 30 sec - exiting 09:24:03 (5448): No heartbeat from core client for 30 sec - exiting 09:24:04 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76ED5EAB read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12372, selfPID=12372, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76ED5EAB read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Sep 2012 10:05:31	927193	15068204	hadcm3n_o650_1980_40_008116599_1	777,600	1,780,718	2.2900
16 Sep 2012 15:51:15	927193	15068204	hadcm3n_o650_1980_40_008116599_1	751,680	1,721,841	2.2907
15 Sep 2012 21:21:24	927193	15068204	hadcm3n_o650_1980_40_008116599_1	725,760	1,663,082	2.2915
15 Sep 2012 04:26:39	927193	15068204	hadcm3n_o650_1980_40_008116599_1	699,840	1,604,885	2.2932
14 Sep 2012 10:17:49	927193	15068204	hadcm3n_o650_1980_40_008116599_1	673,920	1,541,912	2.2880
13 Sep 2012 16:04:06	927193	15068204	hadcm3n_o650_1980_40_008116599_1	648,000	1,478,949	2.2823
12 Sep 2012 03:13:16	927193	15068204	hadcm3n_o650_1980_40_008116599_1	622,080	1,417,109	2.2780
02 Sep 2012 06:35:18	927193	15068204	hadcm3n_o650_1980_40_008116599_1	596,160	1,359,164	2.2799
31 Aug 2012 22:13:03	927193	15068204	hadcm3n_o650_1980_40_008116599_1	570,240	1,300,607	2.2808
30 Aug 2012 07:55:45	927193	15068204	hadcm3n_o650_1980_40_008116599_1	544,320	1,239,886	2.2779
29 Aug 2012 11:36:00	927193	15068204	hadcm3n_o650_1980_40_008116599_1	518,400	1,178,956	2.2742
28 Aug 2012 11:41:40	927193	15068204	hadcm3n_o650_1980_40_008116599_1	492,480	1,118,942	2.2721
27 Aug 2012 07:16:02	927193	15068204	hadcm3n_o650_1980_40_008116599_1	466,560	1,061,467	2.2751
26 Aug 2012 06:19:44	927193	15068204	hadcm3n_o650_1980_40_008116599_1	440,640	1,004,148	2.2788
25 Aug 2012 12:45:32	927193	15068204	hadcm3n_o650_1980_40_008116599_1	414,720	947,765	2.2853
24 Aug 2012 21:27:01	927193	15068204	hadcm3n_o650_1980_40_008116599_1	388,800	888,186	2.2844
23 Aug 2012 22:33:50	927193	15068204	hadcm3n_o650_1980_40_008116599_1	362,880	826,742	2.2783
23 Aug 2012 00:14:28	927193	15068204	hadcm3n_o650_1980_40_008116599_1	336,960	765,172	2.2708
22 Aug 2012 04:15:18	927193	15068204	hadcm3n_o650_1980_40_008116599_1	311,040	704,730	2.2657
20 Aug 2012 06:29:01	927193	15068204	hadcm3n_o650_1980_40_008116599_1	285,120	642,222	2.2525