Task 13734131

Name	hadcm3n_yccr_1940_40_007607934_0
Workunit	7786064
Created	6 Dec 2011, 3:01:31 UTC
Sent	6 Dec 2011, 16:42:43 UTC
Report deadline	7 Mar 2012, 0:09:54 UTC
Received	17 Mar 2012, 13:21:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1041883
Run time	69 days 9 hours 18 min 19 sec
CPU time	43 days 21 hours 22 min 16 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1524, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:20:59 (5180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:22:03 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:15:33 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=904, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4748, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:29:49 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:50 (2172): No heartbeat from core client for 30 sec - exiting 21:29:51 (2172): No heartbeat from core client for 30 sec - exiting 21:29:52 (2172): No heartbeat from core client for 30 sec - exiting 21:29:53 (2172): No heartbeat from core client for 30 sec - exiting 21:29:54 (2172): No heartbeat from core client for 30 sec - exiting 23:24:19 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2808, iMonCtr=1 Model crash detected, will try to restart... 18:06:56 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:57 (5972): No heartbeat from core client for 30 sec - exiting 18:06:58 (5972): No heartbeat from core client for 30 sec - exiting 18:06:59 (5972): No heartbeat from core client for 30 sec - exiting 18:07:00 (5972): No heartbeat from core client for 30 sec - exiting 18:07:01 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:25:00 (5532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:02 (5532): No heartbeat from core client for 30 sec - exiting 13:23:03 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:17:26 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:19 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:20 (2688): No heartbeat from core client for 30 sec - exiting 21:56:15 (1732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:34:33 (1572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:28 (6532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:29 (6532): No heartbeat from core client for 30 sec - exiting 18:40:30 (6532): No heartbeat from core client for 30 sec - exiting 18:40:31 (6532): No heartbeat from core client for 30 sec - exiting 18:40:32 (6532): No heartbeat from core client for 30 sec - exiting 18:40:33 (6532): No heartbeat from core client for 30 sec - exiting 18:40:34 (6532): No heartbeat from core client for 30 sec - exiting 18:49:32 (7316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:33 (7316): No heartbeat from core client for 30 sec - exiting 19:01:51 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:11 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:13 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:23 (7072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:47:43 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:54 (7892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:54:03 (5840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:54:04 (5840): No heartbeat from core client for 30 sec - exiting 19:54:05 (5840): No heartbeat from core client for 30 sec - exiting 19:54:06 (5840): No heartbeat from core client for 30 sec - exiting 19:54:07 (5840): No heartbeat from core client for 30 sec - exiting 19:54:08 (5840): No heartbeat from core client for 30 sec - exiting 19:54:09 (5840): No heartbeat from core client for 30 sec - exiting 20:12:12 (7976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4456, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:12:05 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5652, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10636, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:52 (7924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77456E0F read attempt to address 0x409B0195 Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3320, selfPID=3320, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Mar 2012 05:27:32	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	777,600	3,658,645	4.7050
13 Mar 2012 00:09:07	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	751,680	3,516,588	4.6783
10 Mar 2012 13:32:16	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	725,760	3,385,066	4.6642
08 Mar 2012 00:31:00	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	699,840	3,251,442	4.6460
05 Mar 2012 11:11:39	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	673,920	3,118,633	4.6276
02 Mar 2012 10:02:58	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	648,000	2,974,042	4.5896
28 Feb 2012 22:05:09	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	622,080	2,840,539	4.5662
26 Feb 2012 20:52:45	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	596,160	2,702,009	4.5324
24 Feb 2012 07:37:21	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	570,240	2,571,298	4.5092
22 Feb 2012 03:01:22	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	544,320	2,539,481	4.6654
18 Feb 2012 21:23:37	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	518,400	2,405,001	4.6393
16 Feb 2012 15:00:45	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	492,480	2,272,720	4.6148
14 Feb 2012 14:38:12	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	466,560	2,141,115	4.5892
11 Feb 2012 06:41:36	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	440,640	1,998,794	4.5361
08 Feb 2012 19:50:09	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	414,720	1,852,157	4.4660
06 Feb 2012 15:08:45	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	388,800	1,701,063	4.3752
03 Feb 2012 09:55:05	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	362,880	1,567,463	4.3195
01 Feb 2012 07:38:34	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	336,960	1,457,081	4.3242
28 Jan 2012 13:13:40	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	311,040	1,341,380	4.3126
26 Jan 2012 14:18:48	1041883	13734131	hadcm3n_yccr_1940_40_007607934_0	285,120	1,222,969	4.2893