Task 12732066

Name	hadcm3n_o038_1900_40_007195447_0
Workunit	7393727
Created	28 Mar 2011, 13:55:46 UTC
Sent	3 Apr 2011, 12:21:19 UTC
Report deadline	3 Jul 2011, 19:48:30 UTC
Received	12 Aug 2011, 12:05:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1117362
Run time	16 days 10 hours 0 min 3 sec
CPU time	13 days 21 hours 31 min 55 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3316, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3532, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... 10:55:51 (3768): No heartbeat from core client for 30 sec - exiting 10:55:53 (3768): No heartbeat from core client for 30 sec - exiting 10:55:54 (3768): No heartbeat from core client for 30 sec - exiting 10:55:55 (3768): No heartbeat from core client for 30 sec - exiting 10:55:56 (3768): No heartbeat from core client for 30 sec - exiting 10:55:57 (3768): No heartbeat from core client for 30 sec - exiting 10:55:58 (3768): No heartbeat from core client for 30 sec - exiting 10:55:59 (3768): No heartbeat from core client for 30 sec - exiting 10:56:00 (3768): No heartbeat from core client for 30 sec - exiting 10:56:01 (3768): No heartbeat from core client for 30 sec - exiting 10:56:02 (3768): No heartbeat from core client for 30 sec - exiting 10:56:04 (3768): No heartbeat from core client for 30 sec - exiting 10:56:05 (3768): No heartbeat from core client for 30 sec - exiting 10:56:06 (3768): No heartbeat from core client for 30 sec - exiting 10:56:07 (3768): No heartbeat from core client for 30 sec - exiting 10:56:08 (3768): No heartbeat from core client for 30 sec - exiting 10:56:09 (3768): No heartbeat from core client for 30 sec - exiting 10:56:10 (3768): No heartbeat from core client for 30 sec - exiting 10:56:11 (3768): No heartbeat from core client for 30 sec - exiting 10:56:12 (3768): No heartbeat from core client for 30 sec - exiting 10:56:13 (3768): No heartbeat from core client for 30 sec - exiting 10:56:14 (3768): No heartbeat from core client for 30 sec - exiting 10:56:16 (3768): No heartbeat from core client for 30 sec - exiting 10:56:17 (3768): No heartbeat from core client for 30 sec - exiting 10:56:18 (3768): No heartbeat from core client for 30 sec - exiting 10:56:19 (3768): No heartbeat from core client for 30 sec - exiting 10:56:20 (3768): No heartbeat from core client for 30 sec - exiting 10:56:21 (3768): No heartbeat from core client for 30 sec - exiting 10:56:22 (3768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:56:23 (3768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:22:10 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:26 (1296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3912, iMonCtr=1 Model crash detected, will try to restart... 21:03:31 (2580): No heartbeat from core client for 30 sec - exiting 21:03:33 (2580): No heartbeat from core client for 30 sec - exiting 21:03:34 (2580): No heartbeat from core client for 30 sec - exiting 21:03:35 (2580): No heartbeat from core client for 30 sec - exiting 21:03:36 (2580): No heartbeat from core client for 30 sec - exiting 21:03:37 (2580): No heartbeat from core client for 30 sec - exiting 21:03:38 (2580): No heartbeat from core client for 30 sec - exiting 21:03:39 (2580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3220, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1204, iMonCtr=1 Model crash detected, will try to restart... 18:55:59 (3828): No heartbeat from core client for 30 sec - exiting 18:56:00 (3828): No heartbeat from core client for 30 sec - exiting 18:56:01 (3828): No heartbeat from core client for 30 sec - exiting 18:56:02 (3828): No heartbeat from core client for 30 sec - exiting 18:56:04 (3828): No heartbeat from core client for 30 sec - exiting 18:56:05 (3828): No heartbeat from core client for 30 sec - exiting 18:56:06 (3828): No heartbeat from core client for 30 sec - exiting 18:56:07 (3828): No heartbeat from core client for 30 sec - exiting 18:56:09 (3828): No heartbeat from core client for 30 sec - exiting 18:56:10 (3828): No heartbeat from core client for 30 sec - exiting 18:56:11 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:21 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:23 (4208): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3904, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3412, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... 10:02:38 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:56:09 (5388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:20:28 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:34 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1188, iMonCtr=1 Model crash detected, will try to restart... 06:28:43 (3932): No heartbeat from core client for 30 sec - exiting 06:28:44 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=1 Model crash detected, will try to restart... 20:37:46 (1740): No heartbeat from core client for 30 sec - exiting 20:37:47 (1740): No heartbeat from core client for 30 sec - exiting 20:37:48 (1740): No heartbeat from core client for 30 sec - exiting 20:37:49 (1740): No heartbeat from core client for 30 sec - exiting 20:37:50 (1740): No heartbeat from core client for 30 sec - exiting 20:37:51 (1740): No heartbeat from core client for 30 sec - exiting 20:37:52 (1740): No heartbeat from core client for 30 sec - exiting 20:37:54 (1740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:55 (1740): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5268, iMonCtr=1 Model crash detected, will try to restart... 21:49:14 (4192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:01 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:07 (2280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2288, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... 10:10:48 (4372): No heartbeat from core client for 30 sec - exiting 10:10:50 (4372): No heartbeat from core client for 30 sec - exiting 10:10:51 (4372): No heartbeat from core client for 30 sec - exiting 10:10:52 (4372): No heartbeat from core client for 30 sec - exiting 10:10:53 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4160, iMonCtr=1 Model crash detected, will try to restart... CBUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o038ko.pjb8c10 Error converting file to netcdf: dataout/o038ko.pib8c10 Error converting file to netcdf: dataout/o038ko.pfb8c10 Error converting file to netcdf: dataout/o038ka.phb8c10 Error converting file to netcdf: dataout/o038ka.pgb8c10 Error converting file to netcdf: dataout/o038ka.peb8c10 Error converting file to netcdf: dataout/o038ka.pdb8c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2380, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... C Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77075F1B read attempt to address 0x40A5697E Engaging BOINC Windows Runtime Debugger... Signal 11 received, exiting... Called boinc_finish ERROR: Invalid parameter detected in function (null). File: (null) Line: 0 ERROR: Expression: (null) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Aug 2011 11:05:02	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	518,400	1,199,534	2.3139
08 Aug 2011 10:15:48	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	492,480	1,138,119	2.3110
04 Aug 2011 07:53:36	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	466,560	1,073,572	2.3010
29 Jul 2011 15:08:11	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	440,640	1,010,181	2.2925
25 Jul 2011 19:34:14	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	414,720	950,283	2.2914
25 Jul 2011 15:38:22	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	388,800	890,727	2.2910
07 Jul 2011 16:04:31	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	362,880	829,149	2.2849
05 Jul 2011 05:47:37	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	336,960	764,008	2.2674
27 Jun 2011 01:16:46	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	311,040	704,415	2.2647
20 Jun 2011 00:46:57	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	285,120	644,465	2.2603
11 Jun 2011 07:25:05	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	259,200	584,433	2.2548
05 Jun 2011 03:19:19	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	233,280	526,007	2.2548
28 May 2011 14:56:23	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	207,360	469,491	2.2641
21 May 2011 09:27:08	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	181,440	411,133	2.2659
11 May 2011 15:39:46	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	155,520	351,530	2.2604
07 May 2011 02:48:25	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	129,600	295,430	2.2796
04 May 2011 12:20:12	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	103,680	235,201	2.2685
01 May 2011 14:34:17	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	77,760	180,287	2.3185
21 Apr 2011 13:51:46	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	51,840	121,384	2.3415
12 Apr 2011 13:49:41	1117362	12732066	hadcm3n_o038_1900_40_007195447_0	25,920	60,454	2.3323