Task 15463692

Name	hadcm3n_o5gf_2100_40_008254431_0
Workunit	8409555
Created	26 Nov 2012, 20:10:59 UTC
Sent	26 Nov 2012, 20:11:28 UTC
Report deadline	26 Feb 2013, 3:38:39 UTC
Received	17 Dec 2012, 16:28:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1100141
Run time	10 days 9 hours 17 min
CPU time	8 days 15 hours 13 min 49 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9648, iMonCtr=1 Model crash detected, will try to restart... 18:33:32 (8160): No heartbeat from core client for 30 sec - exiting 18:33:33 (8160): No heartbeat from core client for 30 sec - exiting 18:33:34 (8160): No heartbeat from core client for 30 sec - exiting 18:33:35 (8160): No heartbeat from core client for 30 sec - exiting 18:33:36 (8160): No heartbeat from core client for 30 sec - exiting 18:33:37 (8160): No heartbeat from core client for 30 sec - exiting 18:33:38 (8160): No heartbeat from core client for 30 sec - exiting 18:33:39 (8160): No heartbeat from core client for 30 sec - exiting 18:33:41 (8160): No heartbeat from core client for 30 sec - exiting 18:33:42 (8160): No heartbeat from core client for 30 sec - exiting 18:33:43 (8160): No heartbeat from core client for 30 sec - exiting 18:33:44 (8160): No heartbeat from core client for 30 sec - exiting 18:33:45 (8160): No heartbeat from core client for 30 sec - exiting 18:33:46 (8160): No heartbeat from core client for 30 sec - exiting 18:33:47 (8160): No heartbeat from core client for 30 sec - exiting 18:33:48 (8160): No heartbeat from core client for 30 sec - exiting 18:33:49 (8160): No heartbeat from core client for 30 sec - exiting 18:33:50 (8160): No heartbeat from core client for 30 sec - exiting 18:33:51 (8160): No heartbeat from core client for 30 sec - exiting 18:33:52 (8160): No heartbeat from core client for 30 sec - exiting 18:33:54 (8160): No heartbeat from core client for 30 sec - exiting 18:33:55 (8160): No heartbeat from core client for 30 sec - exiting 18:33:56 (8160): No heartbeat from core client for 30 sec - exiting 18:33:57 (8160): No heartbeat from core client for 30 sec - exiting 18:33:58 (8160): No heartbeat from core client for 30 sec - exiting 18:33:59 (8160): No heartbeat from core client for 30 sec - exiting 18:34:00 (8160): No heartbeat from core client for 30 sec - exiting 18:34:01 (8160): No heartbeat from core client for 30 sec - exiting 18:34:02 (8160): No heartbeat from core client for 30 sec - exiting 18:34:03 (8160): No heartbeat from core client for 30 sec - exiting 18:34:05 (8160): No heartbeat from core client for 30 sec - exiting 18:34:06 (8160): No heartbeat from core client for 30 sec - exiting 18:34:07 (8160): No heartbeat from core client for 30 sec - exiting 18:34:08 (8160): No heartbeat from core client for 30 sec - exiting 18:34:09 (8160): No heartbeat from core client for 30 sec - exiting 18:34:10 (8160): No heartbeat from core client for 30 sec - exiting 18:34:11 (8160): No heartbeat from core client for 30 sec - exiting 18:34:12 (8160): No heartbeat from core client for 30 sec - exiting 18:34:13 (8160): No heartbeat from core client for 30 sec - exiting 18:34:14 (8160): No heartbeat from core client for 30 sec - exiting 18:34:15 (8160): No heartbeat from core client for 30 sec - exiting 18:34:17 (8160): No heartbeat from core client for 30 sec - exiting 18:34:18 (8160): No heartbeat from core client for 30 sec - exiting 18:34:19 (8160): No heartbeat from core client for 30 sec - exiting 18:34:20 (8160): No heartbeat from core client for 30 sec - exiting 18:34:21 (8160): No heartbeat from core client for 30 sec - exiting 18:34:22 (8160): No heartbeat from core client for 30 sec - exiting 18:34:23 (8160): No heartbeat from core client for 30 sec - exiting 18:34:24 (8160): No heartbeat from core client for 30 sec - exiting 18:34:25 (8160): No heartbeat from core client for 30 sec - exiting 18:34:26 (8160): No heartbeat from core client for 30 sec - exiting 18:34:27 (8160): No heartbeat from core client for 30 sec - exiting 18:34:29 (8160): No heartbeat from core client for 30 sec - exiting 18:34:30 (8160): No heartbeat from core client for 30 sec - exiting 18:34:31 (8160): No heartbeat from core client for 30 sec - exiting 18:34:32 (8160): No heartbeat from core client for 30 sec - exiting 18:34:33 (8160): No heartbeat from core client for 30 sec - exiting 18:34:34 (8160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:48:18 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:21 (6844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:25:09 (10524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3228, selfPID=3228, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1844, iMonCtr=1 Model crash detected, will try to restart... 14:05:21 (6632): No heartbeat from core client for 30 sec - exiting 14:05:22 (6632): No heartbeat from core client for 30 sec - exiting 14:05:23 (6632): No heartbeat from core client for 30 sec - exiting 14:05:24 (6632): No heartbeat from core client for 30 sec - exiting 14:05:25 (6632): No heartbeat from core client for 30 sec - exiting 14:05:27 (6632): No heartbeat from core client for 30 sec - exiting 14:05:28 (6632): No heartbeat from core client for 30 sec - exiting 14:05:29 (6632): No heartbeat from core client for 30 sec - exiting 14:05:30 (6632): No heartbeat from core client for 30 sec - exiting 14:05:31 (6632): No heartbeat from core client for 30 sec - exiting 14:05:32 (6632): No heartbeat from core client for 30 sec - exiting 14:05:33 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o5gfko.pjv5c10 Error converting file to netcdf: dataout/o5gfko.piv5c10 Error converting file to netcdf: dataout/o5gfko.pfv5c10 Error converting file to netcdf: dataout/o5gfka.phv5c10 Error converting file to netcdf: dataout/o5gfka.pgv5c10 Error converting file to netcdf: dataout/o5gfka.pev5c10 Error converting file to netcdf: dataout/o5gfka.pdv5c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6720, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is n14:13:55 (7360): No heartbeat from core client for 30 sec - exiting 14:13:56 (7360): No heartbeat from core client for 30 sec - exiting 14:13:57 (7360): No heartbeat from core client for 30 sec - exiting 14:13:58 (7360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o5gfko.pjv5c10 Error converting file to netcdf: dataout/o5gfko.piv5c10 Error converting file to netcdf: dataout/o5gfko.pfv5c10 Error converting file to netcdf: dataout/o5gfka.phv5c10 Error converting file to netcdf: dataout/o5gfka.pgv5c10 Error converting file to netcdf: dataout/o5gfka.pev5c10 Error converting file to netcdf: dataout/o5gfka.pdv5c10 14:25:16 (5796): No heartbeat from core client for 30 sec - exiting 14:25:18 (5796): No heartbeat from core client for 30 sec - exiting 14:25:19 (5796): No heartbeat from core client for 30 sec - exiting 14:25:20 (5796): No heartbeat from core client for 30 sec - exiting 14:25:21 (5796): No heartbeat from core client for 30 sec - exiting 14:25:22 (5796): No heartbeat from core client for 30 sec - exiting 14:25:23 (5796): No heartbeat from core client for 30 sec - exiting 14:25:24 (5796): No heartbeat from core client for 30 sec - exiting 14:25:25 (5796): No heartbeat from core client for 30 sec - exiting 14:25:26 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778E3AB3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Dec 2012 14:40:34	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	518,400	746,018	1.4391
16 Dec 2012 01:03:01	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	492,480	708,932	1.4395
14 Dec 2012 22:56:25	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	466,560	673,027	1.4425
14 Dec 2012 08:09:26	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	440,640	633,653	1.4380
14 Dec 2012 08:09:26	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	414,720	596,752	1.4389
14 Dec 2012 08:09:26	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	388,800	559,309	1.4386
14 Dec 2012 08:09:26	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	362,880	522,265	1.4392
14 Dec 2012 08:09:26	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	336,960	485,309	1.4403
07 Dec 2012 03:51:10	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	311,040	447,560	1.4389
05 Dec 2012 22:16:13	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	285,120	409,960	1.4379
05 Dec 2012 08:44:25	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	259,200	371,532	1.4334
04 Dec 2012 11:22:47	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	233,280	334,009	1.4318
03 Dec 2012 17:26:45	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	207,360	296,789	1.4313
03 Dec 2012 05:58:49	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	181,440	259,932	1.4326
02 Dec 2012 17:25:24	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	155,520	222,098	1.4281
02 Dec 2012 04:02:45	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	129,600	185,097	1.4282
01 Dec 2012 16:17:56	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	103,680	147,697	1.4245
01 Dec 2012 05:00:27	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	77,760	110,626	1.4227
30 Nov 2012 16:44:33	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	51,840	73,640	1.4205
30 Nov 2012 05:26:04	1100141	15463692	hadcm3n_o5gf_2100_40_008254431_0	25,920	37,082	1.4306