Task 13751725

Name	hadcm3n_u5te_1980_40_007458844_4
Workunit	7656347
Created	8 Dec 2011, 11:24:55 UTC
Sent	8 Dec 2011, 11:40:37 UTC
Report deadline	8 Mar 2012, 19:07:48 UTC
Received	5 Jan 2012, 20:44:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1074891
Run time	27 days 17 hours 47 min 26 sec
CPU time	26 days 23 hours 3 min 12 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8116, selfPID=8116, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8296, iMonCtr=1 Model crash detected, will try to restart... 18:26:30 (4656): No heartbeat from core client for 30 sec - exiting 18:26:31 (4656): No heartbeat from core client for 30 sec - exiting 18:26:32 (4656): No heartbeat from core client for 30 sec - exiting 18:26:33 (4656): No heartbeat from core client for 30 sec - exiting 18:26:34 (4656): No heartbeat from core client for 30 sec - exiting 18:26:35 (4656): No heartbeat from core client for 30 sec - exiting 18:26:36 (4656): No heartbeat from core client for 30 sec - exiting 18:26:37 (4656): No heartbeat from core client for 30 sec - exiting 18:26:38 (4656): No heartbeat from core client for 30 sec - exiting 18:26:39 (4656): No heartbeat from core client for 30 sec - exiting 18:26:40 (4656): No heartbeat from core client for 30 sec - exiting 18:26:41 (4656): No heartbeat from core client for 30 sec - exiting 18:26:42 (4656): No heartbeat from core client for 30 sec - exiting 18:26:43 (4656): No heartbeat from core client for 30 sec - exiting 18:26:44 (4656): No heartbeat from core client for 30 sec - exiting 18:26:45 (4656): No heartbeat from core client for 30 sec - exiting 18:26:46 (4656): No heartbeat from core client for 30 sec - exiting 18:26:47 (4656): No heartbeat from core client for 30 sec - exiting 18:26:48 (4656): No heartbeat from core client for 30 sec - exiting 18:26:49 (4656): No heartbeat from core client for 30 sec - exiting 18:26:50 (4656): No heartbeat from core client for 30 sec - exiting 18:26:51 (4656): No heartbeat from core client for 30 sec - exiting 18:26:52 (4656): No heartbeat from core client for 30 sec - exiting 18:26:53 (4656): No heartbeat from core client for 30 sec - exiting 18:26:54 (4656): No heartbeat from core client for 30 sec - exiting 18:26:55 (4656): No heartbeat from core client for 30 sec - exiting 18:26:56 (4656): No heartbeat from core client for 30 sec - exiting 18:26:57 (4656): No heartbeat from core client for 30 sec - exiting 18:26:58 (4656): No heartbeat from core client for 30 sec - exiting 18:26:59 (4656): No heartbeat from core client for 30 sec - exiting 18:27:00 (4656): No heartbeat from core client for 30 sec - exiting 18:27:01 (4656): No heartbeat from core client for 30 sec - exiting 18:27:02 (4656): No heartbeat from core client for 30 sec - exiting 18:27:03 (4656): No heartbeat from core client for 30 sec - exiting 18:27:04 (4656): No heartbeat from core client for 30 sec - exiting 18:27:05 (4656): No heartbeat from core client for 30 sec - exiting 18:27:06 (4656): No heartbeat from core client for 30 sec - exiting 18:27:07 (4656): No heartbeat from core client for 30 sec - exiting 18:27:08 (4656): No heartbeat from core client for 30 sec - exiting 18:27:09 (4656): No heartbeat from core client for 30 sec - exiting 18:27:10 (4656): No heartbeat from core client for 30 sec - exiting 18:27:11 (4656): No heartbeat from core client for 30 sec - exiting 18:27:12 (4656): No heartbeat from core client for 30 sec - exiting 18:27:13 (4656): No heartbeat from core client for 30 sec - exiting 18:27:14 (4656): No heartbeat from core client for 30 sec - exiting 18:27:15 (4656): No heartbeat from core client for 30 sec - exiting 18:27:16 (4656): No heartbeat from core client for 30 sec - exiting 18:27:17 (4656): No heartbeat from core client for 30 sec - exiting 18:27:18 (4656): No heartbeat from core client for 30 sec - exiting 18:27:19 (4656): No heartbeat from core client for 30 sec - exiting 18:27:20 (4656): No heartbeat from core client for 30 sec - exiting 18:27:21 (4656): No heartbeat from core client for 30 sec - exiting 18:27:22 (4656): No heartbeat from core client for 30 sec - exiting 18:27:23 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:14 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:13:18 (3984): No heartbeat from core client for 30 sec - exiting 23:13:19 (3984): No heartbeat from core client for 30 sec - exiting 23:13:20 (3984): No heartbeat from core client for 30 sec - exiting 23:13:21 (3984): No heartbeat from core client for 30 sec - exiting 23:13:22 (3984): No heartbeat from core client for 30 sec - exiting 23:13:23 (3984): No heartbeat from core client for 30 sec - exiting 23:13:24 (3984): No heartbeat from core client for 30 sec - exiting 23:13:25 (3984): No heartbeat from core client for 30 sec - exiting 23:13:26 (3984): No heartbeat from core client for 30 sec - exiting 23:13:27 (3984): No heartbeat from core client for 30 sec - exiting 23:13:28 (3984): No heartbeat from core client for 30 sec - exiting 23:13:29 (3984): No heartbeat from core client for 30 sec - exiting 23:13:30 (3984): No heartbeat from core client for 30 sec - exiting 23:13:31 (3984): No heartbeat from core client for 30 sec - exiting 23:13:32 (3984): No heartbeat from core client for 30 sec - exiting 23:13:33 (3984): No heartbeat from core client for 30 sec - exiting 23:13:34 (3984): No heartbeat from core client for 30 sec - exiting 23:13:35 (3984): No heartbeat from core client for 30 sec - exiting 23:13:36 (3984): No heartbeat from core client for 30 sec - exiting 23:13:37 (3984): No heartbeat from core client for 30 sec - exiting 23:13:38 (3984): No heartbeat from core client for 30 sec - exiting 23:13:39 (3984): No heartbeat from core client for 30 sec - exiting 23:13:40 (3984): No heartbeat from core client for 30 sec - exiting 23:13:41 (3984): No heartbeat from core client for 30 sec - exiting 23:13:42 (3984): No heartbeat from core client for 30 sec - exiting 23:13:43 (3984): No heartbeat from core client for 30 sec - exiting 23:13:44 (3984): No heartbeat from core client for 30 sec - exiting 23:13:45 (3984): No heartbeat from core client for 30 sec - exiting 23:13:46 (3984): No heartbeat from core client for 30 sec - exiting 23:13:47 (3984): No heartbeat from core client for 30 sec - exiting 23:13:48 (3984): No heartbeat from core client for 30 sec - exiting 23:13:49 (3984): No heartbeat from core client for 30 sec - exiting 23:13:50 (3984): No heartbeat from core client for 30 sec - exiting 23:13:51 (3984): No heartbeat from core client for 30 sec - exiting 23:13:52 (3984): No heartbeat from core client for 30 sec - exiting 23:13:53 (3984): No heartbeat from core client for 30 sec - exiting 23:13:54 (3984): No heartbeat from core client for 30 sec - exiting 23:13:55 (3984): No heartbeat from core client for 30 sec - exiting 23:13:56 (3984): No heartbeat from core client for 30 sec - exiting 23:13:57 (3984): No heartbeat from core client for 30 sec - exiting 23:13:58 (3984): No heartbeat from core client for 30 sec - exiting 23:13:59 (3984): No heartbeat from core client for 30 sec - exiting 23:14:00 (3984): No heartbeat from core client for 30 sec - exiting 23:14:01 (3984): No heartbeat from core client for 30 sec - exiting 23:14:02 (3984): No heartbeat from core client for 30 sec - exiting 23:14:03 (3984): No heartbeat from core client for 30 sec - exiting 23:14:04 (3984): No heartbeat from core client for 30 sec - exiting 23:14:05 (3984): No heartbeat from core client for 30 sec - exiting 23:14:06 (3984): No heartbeat from core client for 30 sec - exiting 23:14:07 (3984): No heartbeat from core client for 30 sec - exiting 23:14:08 (3984): No heartbeat from core client for 30 sec - exiting 23:14:09 (3984): No heartbeat from core client for 30 sec - exiting 23:14:10 (3984): No heartbeat from core client for 30 sec - exiting 23:14:11 (3984): No heartbeat from core client for 30 sec - exiting 23:14:12 (3984): No heartbeat from core client for 30 sec - exiting 23:14:13 (3984): No heartbeat from core client for 30 sec - exiting 23:14:14 (3984): No heartbeat from core client for 30 sec - exiting 23:14:15 (3984): No heartbeat from core client for 30 sec - exiting 23:14:16 (3984): No heartbeat from core client for 30 sec - exiting 23:14:17 (3984): No heartbeat from core client for 30 sec - exiting 23:14:18 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:07 (5604): No heartbeat from core client for 30 sec - exiting 23:27:08 (5604): No heartbeat from core client for 30 sec - exiting 23:27:09 (5604): No heartbeat from core client for 30 sec - exiting 23:27:10 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:27:34 (8136): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77BD36F9 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_u5te_1980_40_007458844/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jan 2012 19:44:07	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	1,036,800	2,329,383	2.2467
05 Jan 2012 14:30:56	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	1,010,880	2,271,960	2.2475
04 Jan 2012 10:51:40	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	984,960	2,214,155	2.2480
03 Jan 2012 19:09:59	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	959,040	2,155,439	2.2475
03 Jan 2012 04:35:15	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	933,120	2,099,412	2.2499
02 Jan 2012 10:00:03	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	907,200	2,044,075	2.2532
01 Jan 2012 18:19:38	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	881,280	1,985,346	2.2528
31 Dec 2011 23:50:22	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	855,360	1,926,585	2.2524
31 Dec 2011 07:41:11	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	829,440	1,868,429	2.2526
30 Dec 2011 14:22:41	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	803,520	1,809,867	2.2524
29 Dec 2011 21:41:18	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	777,600	1,751,000	2.2518
28 Dec 2011 22:15:35	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	751,680	1,692,713	2.2519
28 Dec 2011 05:44:06	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	725,760	1,634,635	2.2523
27 Dec 2011 13:08:31	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	699,840	1,576,128	2.2521
26 Dec 2011 21:24:22	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	673,920	1,518,178	2.2528
26 Dec 2011 04:09:18	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	648,000	1,459,705	2.2526
25 Dec 2011 11:39:31	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	622,080	1,401,418	2.2528
24 Dec 2011 20:10:15	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	596,160	1,343,468	2.2535
24 Dec 2011 03:23:58	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	570,240	1,285,341	2.2540
23 Dec 2011 11:11:47	1074891	13751725	hadcm3n_u5te_1980_40_007458844_4	544,320	1,227,544	2.2552