Task 15602222

Name	hadcm3n_4d5d_1940_40_008310304_2
Workunit	8461439
Created	9 Feb 2013, 6:57:07 UTC
Sent	9 Feb 2013, 7:09:37 UTC
Report deadline	11 May 2013, 14:36:48 UTC
Received	17 Apr 2013, 0:52:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1268383
Run time	10 days 20 hours 37 min 57 sec
CPU time	10 days 11 hours 39 min 57 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7256, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6508, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6780, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:37:34 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:35 (6608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:02:44 (6636): No heartbeat from core client for 30 sec - exiting 01:02:45 (6636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6352, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10496, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7088, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6348, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:13:56 (6752): No heartbeat from core client for 30 sec - exiting 20:13:57 (6752): No heartbeat from core client for 30 sec - exiting 20:13:58 (6752): No heartbeat from core client for 30 sec - exiting 20:13:59 (6752): No heartbeat from core client for 30 sec - exiting 20:14:00 (6752): No heartbeat from core client for 30 sec - exiting 20:14:01 (6752): No heartbeat from core client for 30 sec - exiting 20:14:02 (6752): No heartbeat from core client for 30 sec - exiting 20:14:03 (6752): No heartbeat from core client for 30 sec - exiting 20:14:04 (6752): No heartbeat from core client for 30 sec - exiting 20:14:05 (6752): No heartbeat from core client for 30 sec - exiting 20:14:06 (6752): No heartbeat from core client for 30 sec - exiting 20:14:07 (6752): No heartbeat from core client for 30 sec - exiting 20:14:08 (6752): No heartbeat from core client for 30 sec - exiting 20:14:09 (6752): No heartbeat from core client for 30 sec - exiting 20:14:10 (6752): No heartbeat from core client for 30 sec - exiting 20:14:11 (6752): No heartbeat from core client for 30 sec - exiting 20:14:12 (6752): No heartbeat from core client for 30 sec - exiting 20:14:13 (6752): No heartbeat from core client for 30 sec - exiting 20:14:14 (6752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:15 (6752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77167373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x773C7373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_4d5d_1940_40_008310304/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Apr 2013 15:56:30	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	777,600	901,230	1.1590
12 Apr 2013 02:27:07	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	751,680	873,443	1.1620
10 Apr 2013 14:25:23	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	725,760	848,515	1.1691
08 Apr 2013 22:15:57	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	699,840	823,785	1.1771
05 Apr 2013 20:15:53	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	673,920	796,482	1.1819
05 Apr 2013 12:26:21	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	648,000	768,879	1.1865
04 Apr 2013 21:19:59	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	622,080	739,315	1.1885
02 Apr 2013 17:01:29	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	596,160	713,743	1.1972
30 Mar 2013 22:00:53	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	570,240	687,803	1.2062
29 Mar 2013 15:40:41	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	544,320	659,961	1.2125
26 Mar 2013 12:17:18	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	518,400	636,801	1.2284
25 Mar 2013 16:52:06	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	492,480	606,455	1.2314
24 Mar 2013 23:44:52	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	466,560	576,016	1.2346
24 Mar 2013 01:49:46	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	440,640	547,639	1.2428
22 Mar 2013 03:17:15	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	414,720	523,500	1.2623
21 Mar 2013 18:11:19	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	388,800	501,030	1.2887
21 Mar 2013 13:00:17	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	362,880	482,313	1.3291
20 Mar 2013 23:06:27	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	336,960	458,918	1.3619
17 Mar 2013 02:57:42	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	311,040	430,487	1.3840
13 Mar 2013 19:05:03	1268383	15602222	hadcm3n_4d5d_1940_40_008310304_2	285,120	404,041	1.4171