Task 15598381

Name	hadcm3n_4asq_1940_40_008310921_0
Workunit	8462056
Created	8 Feb 2013, 2:38:03 UTC
Sent	9 Feb 2013, 0:19:41 UTC
Report deadline	11 May 2013, 7:46:52 UTC
Received	29 Apr 2013, 18:22:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1145857
Run time	11 days 17 hours 25 min 13 sec
CPU time	9 days 22 hours 7 min 39 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 19:34:08 (2340): No heartbeat from core client for 30 sec - exiting 19:34:09 (2340): No heartbeat from core client for 30 sec - exiting 19:34:10 (2340): No heartbeat from core client for 30 sec - exiting 19:34:11 (2340): No heartbeat from core client for 30 sec - exiting 19:34:12 (2340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:03:57 (5156): No heartbeat from core client for 30 sec - exiting 15:03:58 (5156): No heartbeat from core client for 30 sec - exiting 15:03:59 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:14:23 (2764): No heartbeat from core client for 30 sec - exiting 09:14:24 (2764): No heartbeat from core client for 30 sec - exiting 09:14:25 (2764): No heartbeat from core client for 30 sec - exiting 09:14:26 (2764): No heartbeat from core client for 30 sec - exiting 09:14:27 (2764): No heartbeat from core client for 30 sec - exiting 09:14:28 (2764): No heartbeat from core client for 30 sec - exiting 09:14:29 (2764): No heartbeat from core client for 30 sec - exiting 09:14:31 (2764): No heartbeat from core client for 30 sec - exiting 09:14:32 (2764): No heartbeat from core client for 30 sec - exiting 09:14:33 (2764): No heartbeat from core client for 30 sec - exiting 09:14:34 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:35 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2064, iMonCtr=1 Model crash detected, will try to restart... 23:41:44 (928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:48:20 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:12:12 (5012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4956, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 18:09:18 (2536): No heartbeat from core client for 30 sec - exiting 18:09:19 (2536): No heartbeat from core client for 30 sec - exiting 18:09:20 (2536): No heartbeat from core client for 30 sec - exiting 18:09:21 (2536): No heartbeat from core client for 30 sec - exiting 18:09:22 (2536): No heartbeat from core client for 30 sec - exiting 18:09:23 (2536): No heartbeat from core client for 30 sec - exiting 18:09:24 (2536): No heartbeat from core client for 30 sec - exiting 18:09:25 (2536): No heartbeat from core client for 30 sec - exiting 18:09:27 (2536): No heartbeat from core client for 30 sec - exiting 18:09:28 (2536): No heartbeat from core client for 30 sec - exiting 18:09:29 (2536): No heartbeat from core client for 30 sec - exiting 18:09:30 (2536): No heartbeat from core client for 30 sec - exiting 18:09:31 (2536): No heartbeat from core client for 30 sec - exiting 18:09:32 (2536): No heartbeat from core client for 30 sec - exiting 18:09:33 (2536): No heartbeat from core client for 30 sec - exiting 18:09:34 (2536): No heartbeat from core client for 30 sec - exiting 18:09:35 (2536): No heartbeat from core client for 30 sec - exiting 18:09:36 (2536): No heartbeat from core client for 30 sec - exiting 18:09:37 (2536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:12:42 (4900): No heartbeat from core client for 30 sec - exiting 00:12:43 (4900): No heartbeat from core client for 30 sec - exiting 00:12:44 (4900): No heartbeat from core client for 30 sec - exiting 00:12:46 (4900): No heartbeat from core client for 30 sec - exiting 00:12:47 (4900): No heartbeat from core client for 30 sec - exiting 00:12:48 (4900): No heartbeat from core client for 30 sec - exiting 00:12:49 (4900): No heartbeat from core client for 30 sec - exiting 00:12:50 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:51 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:50:37 (2308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:44:05 (5872): No heartbeat from core client for 30 sec - exiting 14:44:06 (5872): No heartbeat from core client for 30 sec - exiting 14:44:07 (5872): No heartbeat from core client for 30 sec - exiting 14:44:08 (5872): No heartbeat from core client for 30 sec - exiting 14:44:09 (5872): No heartbeat from core client for 30 sec - exiting 14:44:10 (5872): No heartbeat from core client for 30 sec - exiting 14:44:11 (5872): No heartbeat from core client for 30 sec - exiting 14:44:12 (5872): No heartbeat from core client for 30 sec - exiting 14:44:13 (5872): No heartbeat from core client for 30 sec - exiting 14:44:14 (5872): No heartbeat from core client for 30 sec - exiting 14:44:15 (5872): No heartbeat from core client for 30 sec - exiting 14:44:16 (5872): No heartbeat from core client for 30 sec - exiting 14:44:17 (5872): No heartbeat from core client for 30 sec - exiting 14:44:18 (5872): No heartbeat from core client for 30 sec - exiting 14:44:19 (5872): No heartbeat from core client for 30 sec - exiting 14:44:20 (5872): No heartbeat from core client for 30 sec - exiting 14:44:21 (5872): No heartbeat from core client for 30 sec - exiting 14:44:22 (5872): No heartbeat from core client for 30 sec - exiting 14:44:23 (5872): No heartbeat from core client for 30 sec - exiting 14:44:24 (5872): No heartbeat from core client for 30 sec - exiting 14:44:25 (5872): No heartbeat from core client for 30 sec - exiting 14:44:26 (5872): No heartbeat from core client for 30 sec - exiting 14:44:27 (5872): No heartbeat from core client for 30 sec - exiting 14:44:28 (5872): No heartbeat from core client for 30 sec - exiting 14:44:29 (5872): No heartbeat from core client for 30 sec - exiting 14:44:30 (5872): No heartbeat from core client for 30 sec - exiting 14:44:31 (5872): No heartbeat from core client for 30 sec - exiting 14:44:32 (5872): No heartbeat from core client for 30 sec - exiting 14:44:33 (5872): No heartbeat from core client for 30 sec - exiting 14:44:34 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:46:25 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=208, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 10:08:07 (3552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... 17:56:49 (2812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:12:56 (2092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:35:12 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:07:49 (5928): No heartbeat from core client for 30 sec - exiting 20:07:50 (5928): No heartbeat from core client for 30 sec - exiting 20:07:51 (5928): No heartbeat from core client for 30 sec - exiting 20:07:52 (5928): No heartbeat from core client for 30 sec - exiting 20:07:53 (5928): No heartbeat from core client for 30 sec - exiting 20:07:54 (5928): No heartbeat from core client for 30 sec - exiting 20:07:55 (5928): No heartbeat from core client for 30 sec - exiting 20:07:56 (5928): No heartbeat from core client for 30 sec - exiting 20:07:57 (5928): No heartbeat from core client for 30 sec - exiting 20:07:58 (5928): No heartbeat from core client for 30 sec - exiting 20:07:59 (5928): No heartbeat from core client for 30 sec - exiting 20:08:00 (5928): No heartbeat from core client for 30 sec - exiting 20:08:01 (5928): No heartbeat from core client for 30 sec - exiting 20:08:02 (5928): No heartbeat from core client for 30 sec - exiting 20:08:03 (5928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1196, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:25:39 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5620, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:03:20 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:03:22 (3840): No heartbeat from core client for 30 sec - exiting 00:03:23 (3840): No heartbeat from core client for 30 sec - exiting 00:03:24 (3840): No heartbeat from core client for 30 sec - exiting 00:03:25 (3840): No heartbeat from core client for 30 sec - exiting 00:03:26 (3840): No heartbeat from core client for 30 sec - exiting 00:03:27 (3840): No heartbeat from core client for 30 sec - exiting 00:04:17 (4872): No heartbeat from core client for 30 sec - exiting 00:04:18 (4872): No heartbeat from core client for 30 sec - exiting 00:04:19 (4872): No heartbeat from core client for 30 sec - exiting 00:04:20 (4872): No heartbeat from core client for 30 sec - exiting 00:04:21 (4872): No heartbeat from core client for 30 sec - exiting 00:04:22 (4872): No heartbeat from core client for 30 sec - exiting 00:04:23 (4872): No heartbeat from core client for 30 sec - exiting 00:04:24 (4872): No heartbeat from core client for 30 sec - exiting 00:04:25 (4872): No heartbeat from core client for 30 sec - exiting 00:04:26 (4872): No heartbeat from core client for 30 sec - exiting 00:04:27 (4872): No heartbeat from core client for 30 sec - exiting 00:04:28 (4872): No heartbeat from core client for 30 sec - exiting 00:04:29 (4872): No heartbeat from core client for 30 sec - exiting 00:04:30 (4872): No heartbeat from core client for 30 sec - exiting 00:04:31 (4872): No heartbeat from core client for 30 sec - exiting 00:04:32 (4872): No heartbeat from core client for 30 sec - exiting 00:04:33 (4872): No heartbeat from core client for 30 sec - exiting 00:04:34 (4872): No heartbeat from core client for 30 sec - exiting 00:04:35 (4872): No heartbeat from core client for 30 sec - exiting 00:04:36 (4872): No heartbeat from core client for 30 sec - exiting 00:04:37 (4872): No heartbeat from core client for 30 sec - exiting 00:04:38 (4872): No heartbeat from core client for 30 sec - exiting 00:04:39 (4872): No heartbeat from core client for 30 sec - exiting 00:04:40 (4872): No heartbeat from core client for 30 sec - exiting 00:04:41 (4872): No heartbeat from core client for 30 sec - exiting 00:04:42 (4872): No heartbeat from core client for 30 sec - exiting 00:04:43 (4872): No heartbeat from core client for 30 sec - exiting 00:04:44 (4872): No heartbeat from core client for 30 sec - exiting 00:04:45 (4872): No heartbeat from core client for 30 sec - exiting 00:04:46 (4872): No heartbeat from core client for 30 sec - exiting 00:04:47 (4872): No heartbeat from core client for 30 sec - exiting 00:04:48 (4872): No heartbeat from core client for 30 sec - exiting 00:04:49 (4872): No heartbeat from core client for 30 sec - exiting 00:04:50 (4872): No heartbeat from core client for 30 sec - exiting 00:04:51 (4872): No heartbeat from core client for 30 sec - exiting 00:04:52 (4872): No heartbeat from core client for 30 sec - exiting 00:04:53 (4872): No heartbeat from core client for 30 sec - exiting 00:04:54 (4872): No heartbeat from core client for 30 sec - exiting 00:04:55 (4872): No heartbeat from core client for 30 sec - exiting 00:04:56 (4872): No heartbeat from core client for 30 sec - exiting 00:04:57 (4872): No heartbeat from core client for 30 sec - exiting 00:04:58 (4872): No heartbeat from core client for 30 sec - exiting 00:04:59 (4872): No heartbeat from core client for 30 sec - exiting 00:05:00 (4872): No heartbeat from core client for 30 sec - exiting 00:05:01 (4872): No heartbeat from core client for 30 sec - exiting 00:05:02 (4872): No heartbeat from core client for 30 sec - exiting 00:05:03 (4872): No heartbeat from core client for 30 sec - exiting 00:05:04 (4872): No heartbeat from core client for 30 sec - exiting 00:05:05 (4872): No heartbeat from core client for 30 sec - exiting 00:05:06 (4872): No heartbeat from core client for 30 sec - exiting 00:05:07 (4872): No heartbeat from core client for 30 sec - exiting 00:05:08 (4872): No heartbeat from core client for 30 sec - exiting 00:05:09 (4872): No heartbeat from core client for 30 sec - exiting 00:05:10 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:11 (4872): No heartbeat from core client for 30 sec - exiting 00:05:12 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:23:56 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:21:16 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:47:28 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B52AB1 read attempt to address 0x405941DD Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F62AB1 read attempt to address 0x405941DD Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_4asq_1940_40_008310921/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Apr 2013 14:55:41	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	259,200	834,705	3.2203
13 Apr 2013 23:48:18	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	233,280	754,887	3.2360
02 Apr 2013 16:51:26	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	207,360	670,017	3.2312
27 Mar 2013 18:37:13	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	181,440	584,370	3.2207
24 Mar 2013 22:39:27	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	155,520	498,767	3.2071
17 Mar 2013 19:04:49	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	129,600	414,250	3.1964
11 Mar 2013 18:39:18	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	103,680	329,780	3.1807
02 Mar 2013 15:15:40	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	77,760	245,939	3.1628
19 Feb 2013 19:37:39	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	51,840	163,485	3.1536
15 Feb 2013 19:38:45	1145857	15598381	hadcm3n_4asq_1940_40_008310921_0	25,920	81,350	3.1385