Task 16066068

Name	hadcm3n_47rf_2020_40_008399382_2
Workunit	8550238
Created	11 Oct 2013, 22:28:01 UTC
Sent	11 Oct 2013, 22:28:18 UTC
Report deadline	11 Jan 2014, 5:55:29 UTC
Received	20 Jan 2014, 17:30:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1063824
Run time	17 days 2 hours 14 min 39 sec
CPU time	15 days 9 hours 55 min 51 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:48:26 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:34:01 (2300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:23 (2300): No heartbeat from core client for 30 sec - exiting 18:34:24 (2300): No heartbeat from core client for 30 sec - exiting 18:34:25 (2300): No heartbeat from core client for 30 sec - exiting 18:34:26 (2300): No heartbeat from core client for 30 sec - exiting 18:34:27 (2300): No heartbeat from core client for 30 sec - exiting 18:34:28 (2300): No heartbeat from core client for 30 sec - exiting 18:34:29 (2300): No heartbeat from core client for 30 sec - exiting 18:34:30 (2300): No heartbeat from core client for 30 sec - exiting 18:34:31 (2300): No heartbeat from core client for 30 sec - exiting 18:34:32 (2300): No heartbeat from core client for 30 sec - exiting 18:34:34 (2300): No heartbeat from core client for 30 sec - exiting 18:34:35 (2300): No heartbeat from core client for 30 sec - exiting 18:34:36 (2300): No heartbeat from core client for 30 sec - exiting 18:34:37 (2300): No heartbeat from core client for 30 sec - exiting 18:34:38 (2300): No heartbeat from core client for 30 sec - exiting 18:34:39 (2300): No heartbeat from core client for 30 sec - exiting 18:34:40 (2300): No heartbeat from core client for 30 sec - exiting 18:34:41 (2300): No heartbeat from core client for 30 sec - exiting 18:34:42 (2300): No heartbeat from core client for 30 sec - exiting 18:34:43 (2300): No heartbeat from core client for 30 sec - exiting 18:34:44 (2300): No heartbeat from core client for 30 sec - exiting 18:34:46 (2300): No heartbeat from core client for 30 sec - exiting 18:34:47 (2300): No heartbeat from core client for 30 sec - exiting 18:34:48 (2300): No heartbeat from core client for 30 sec - exiting 18:34:49 (2300): No heartbeat from core client for 30 sec - exiting 18:34:50 (2300): No heartbeat from core client for 30 sec - exiting 18:34:51 (2300): No heartbeat from core client for 30 sec - exiting 18:34:52 (2300): No heartbeat from core client for 30 sec - exiting 18:34:53 (2300): No heartbeat from core client for 30 sec - exiting 18:34:54 (2300): No heartbeat from core client for 30 sec - exiting 18:34:55 (2300): No heartbeat from core client for 30 sec - exiting 18:34:56 (2300): No heartbeat from core client for 30 sec - exiting 18:34:58 (2300): No heartbeat from core client for 30 sec - exiting 18:34:59 (2300): No heartbeat from core client for 30 sec - exiting 18:35:00 (2300): No heartbeat from core client for 30 sec - exiting 18:35:01 (2300): No heartbeat from core client for 30 sec - exiting 18:35:02 (2300): No heartbeat from core client for 30 sec - exiting 18:35:03 (2300): No heartbeat from core client for 30 sec - exiting 18:35:04 (2300): No heartbeat from core client for 30 sec - exiting 18:35:05 (2300): No heartbeat from core client for 30 sec - exiting 18:35:06 (2300): No heartbeat from core client for 30 sec - exiting 18:35:10 (2300): No heartbeat from core client for 30 sec - exiting 18:35:11 (2300): No heartbeat from core client for 30 sec - exiting 18:35:12 (2300): No heartbeat from core client for 30 sec - exiting 18:35:13 (2300): No heartbeat from core client for 30 sec - exiting 18:35:14 (2300): No heartbeat from core client for 30 sec - exiting 18:35:15 (2300): No heartbeat from core client for 30 sec - exiting 18:35:16 (2300): No heartbeat from core client for 30 sec - exiting 18:35:17 (2300): No heartbeat from core client for 30 sec - exiting 18:35:18 (2300): No heartbeat from core client for 30 sec - exiting 18:35:20 (2300): No heartbeat from core client for 30 sec - exiting 18:35:21 (2300): No heartbeat from core client for 30 sec - exiting 18:35:22 (2300): No heartbeat from core client for 30 sec - exiting 18:35:23 (2300): No heartbeat from core client for 30 sec - exiting 18:35:24 (2300): No heartbeat from core client for 30 sec - exiting 18:35:25 (2300): No heartbeat from core client for 30 sec - exiting 18:35:26 (2300): No heartbeat from core client for 30 sec - exiting 18:35:27 (2300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:30:56 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:31:41 (4772): No heartbeat from core client for 30 sec - exiting 12:31:42 (4772): No heartbeat from core client for 30 sec - exiting 12:31:43 (4772): No heartbeat from core client for 30 sec - exiting 12:31:44 (4772): No heartbeat from core client for 30 sec - exiting 12:31:45 (4772): No heartbeat from core client for 30 sec - exiting 12:31:46 (4772): No heartbeat from core client for 30 sec - exiting 12:31:47 (4772): No heartbeat from core client for 30 sec - exiting 12:31:48 (4772): No heartbeat from core client for 30 sec - exiting 12:31:49 (4772): No heartbeat from core client for 30 sec - exiting 12:31:50 (4772): No heartbeat from core client for 30 sec - exiting 12:31:52 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:49:20 (1620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:44 (3544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:04 (3760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:07 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:08 (5864): No heartbeat from core client for 30 sec - exiting 17:20:09 (5864): No heartbeat from core client for 30 sec - exiting 17:20:10 (5864): No heartbeat from core client for 30 sec - exiting 17:20:11 (5864): No heartbeat from core client for 30 sec - exiting 17:20:12 (5864): No heartbeat from core client for 30 sec - exiting 17:20:13 (5864): No heartbeat from core client for 30 sec - exiting 18:17:25 (2320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:06 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:07 (2720): No heartbeat from core client for 30 sec - exiting 19:24:08 (2720): No heartbeat from core client for 30 sec - exiting 19:24:09 (2720): No heartbeat from core client for 30 sec - exiting 19:24:10 (2720): No heartbeat from core client for 30 sec - exiting 19:27:13 (6320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:19 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:21 (7096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:56 (7164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:56 (7072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:04:11 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:17 (3524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:13:21 (1800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:14:01 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:26:25 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:37:04 (2004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:15 (7060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6736, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x770AFF6B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jan 2014 16:25:08	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	1,036,800	1,331,334	1.2841
17 Jan 2014 19:23:50	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	1,010,880	1,297,746	1.2838
15 Jan 2014 20:20:26	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	984,960	1,264,276	1.2836
15 Jan 2014 12:55:04	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	959,040	1,231,357	1.2839
10 Jan 2014 22:17:42	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	933,120	1,198,039	1.2839
10 Jan 2014 17:05:10	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	907,200	1,164,999	1.2842
08 Jan 2014 07:35:28	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	881,280	1,131,629	1.2841
06 Jan 2014 23:10:23	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	855,360	1,104,838	1.2917
04 Jan 2014 23:12:46	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	829,440	1,071,631	1.2920
03 Jan 2014 20:01:11	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	803,520	1,038,627	1.2926
30 Dec 2013 12:27:51	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	777,600	1,005,297	1.2928
27 Dec 2013 18:32:00	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	751,680	972,530	1.2938
26 Dec 2013 19:08:26	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	725,760	939,776	1.2949
25 Dec 2013 16:43:28	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	699,840	906,367	1.2951
20 Dec 2013 16:30:52	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	673,920	873,002	1.2954
16 Dec 2013 13:59:25	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	648,000	839,999	1.2963
14 Dec 2013 16:50:38	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	622,080	806,964	1.2972
13 Dec 2013 18:40:56	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	596,160	774,155	1.2986
10 Dec 2013 22:54:53	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	570,240	740,199	1.2980
04 Dec 2013 18:11:55	1063824	16066068	hadcm3n_47rf_2020_40_008399382_2	544,320	706,785	1.2985