Task 13108716

Name	hadcm3n_yf3h_1900_40_007352423_1
Workunit	7549853
Created	6 Jul 2011, 14:20:26 UTC
Sent	15 Jul 2011, 22:58:36 UTC
Report deadline	15 Oct 2011, 6:25:47 UTC
Received	17 Aug 2011, 23:33:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1065973
Run time	5 days 16 hours 57 min 21 sec
CPU time	5 days 16 hours 57 min 21 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.61 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 18:49:43 (4320): No heartbeat from core client for 30 sec - exiting 18:49:44 (4320): No heartbeat from core client for 30 sec - exiting 18:49:45 (4320): No heartbeat from core client for 30 sec - exiting 18:49:46 (4320): No heartbeat from core client for 30 sec - exiting 18:49:47 (4320): No heartbeat from core client for 30 sec - exiting 18:49:48 (4320): No heartbeat from core client for 30 sec - exiting 18:49:49 (4320): No heartbeat from core client for 30 sec - exiting 18:49:50 (4320): No heartbeat from core client for 30 sec - exiting 18:49:51 (4320): No heartbeat from core client for 30 sec - exiting 18:49:53 (4320): No heartbeat from core client for 30 sec - exiting 18:49:54 (4320): No heartbeat from core client for 30 sec - exiting 18:49:55 (4320): No heartbeat from core client for 30 sec - exiting 18:49:56 (4320): No heartbeat from core client for 30 sec - exiting 18:49:57 (4320): No heartbeat from core client for 30 sec - exiting 18:49:58 (4320): No heartbeat from core client for 30 sec - exiting 18:49:59 (4320): No heartbeat from core client for 30 sec - exiting 18:50:00 (4320): No heartbeat from core client for 30 sec - exiting 18:50:01 (4320): No heartbeat from core client for 30 sec - exiting 18:50:02 (4320): No heartbeat from core client for 30 sec - exiting 18:50:03 (4320): No heartbeat from core client for 30 sec - exiting 18:50:05 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:06 (4320): No heartbeat from core client for 30 sec - exiting 18:50:07 (4320): No heartbeat from core client for 30 sec - exiting 18:50:08 (4320): No heartbeat from core client for 30 sec - exiting 18:50:09 (4320): No heartbeat from core client for 30 sec - exiting 18:50:10 (4320): No heartbeat from core client for 30 sec - exiting 18:50:11 (4320): No heartbeat from core client for 30 sec - exiting 18:50:12 (4320): No heartbeat from core client for 30 sec - exiting 18:50:13 (4320): No heartbeat from core client for 30 sec - exiting 18:50:14 (4320): No heartbeat from core client for 30 sec - exiting 18:50:15 (4320): No heartbeat from core client for 30 sec - exiting C07:38:45 (2380): No heartbeat from core client for 30 sec - exiting 07:38:46 (2380): No heartbeat from core client for 30 sec - exiting 07:38:47 (2380): No heartbeat from core client for 30 sec - exiting 07:38:48 (2380): No heartbeat from core client for 30 sec - exiting 07:38:49 (2380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:48 (3956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:48 (1508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:07 (880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:14 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:58 (4004): No heartbeat from core client for 30 sec - exiting 07:15:59 (4004): No heartbeat from core client for 30 sec - exiting 07:16:01 (4004): No heartbeat from core client for 30 sec - exiting 07:16:02 (4004): No heartbeat from core client for 30 sec - exiting 07:16:03 (4004): No heartbeat from core client for 30 sec - exiting 07:16:04 (4004): No heartbeat from core client for 30 sec - exiting 07:16:05 (4004): No heartbeat from core client for 30 sec - exiting 07:16:06 (4004): No heartbeat from core client for 30 sec - exiting 07:16:07 (4004): No heartbeat from core client for 30 sec - exiting 07:16:08 (4004): No heartbeat from core client for 30 sec - exiting 07:16:09 (4004): No heartbeat from core client for 30 sec - exiting 07:16:10 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:46:59 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:29 (4228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:35 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3564, iMonCtr=1 Model crash detected, will try to restart... 19:52:33 (128): No heartbeat from core client for 30 sec - exiting 19:52:34 (128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:35 (128): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:11:55 (4068): No heartbeat from core client for 30 sec - exiting 19:11:56 (4068): No heartbeat from core client for 30 sec - exiting 19:11:57 (4068): No heartbeat from core client for 30 sec - exiting 19:11:58 (4068): No heartbeat from core client for 30 sec - exiting 19:11:59 (4068): No heartbeat from core client for 30 sec - exiting 19:12:00 (4068): No heartbeat from core client for 30 sec - exiting 19:12:01 (4068): No heartbeat from core client for 30 sec - exiting 19:12:02 (4068): No heartbeat from core client for 30 sec - exiting 19:12:03 (4068): No heartbeat from core client for 30 sec - exiting 19:12:04 (4068): No heartbeat from core client for 30 sec - exiting 19:12:06 (4068): No heartbeat from core client for 30 sec - exiting 19:12:07 (4068): No heartbeat from core client for 30 sec - exiting 19:12:08 (4068): No heartbeat from core client for 30 sec - exiting 19:12:09 (4068): No heartbeat from core client for 30 sec - exiting 19:12:10 (4068): No heartbeat from core client for 30 sec - exiting 19:12:11 (4068): No heartbeat from core client for 30 sec - exiting 19:12:12 (4068): No heartbeat from core client for 30 sec - exiting 19:12:13 (4068): No heartbeat from core client for 30 sec - exiting 19:12:14 (4068): No heartbeat from core client for 30 sec - exiting 19:12:15 (4068): No heartbeat from core client for 30 sec - exiting 19:12:16 (4068): No heartbeat from core client for 30 sec - exiting 19:12:18 (4068): No heartbeat from core client for 30 sec - exiting 19:12:19 (4068): No heartbeat from core client for 30 sec - exiting 19:12:20 (4068): No heartbeat from core client for 30 sec - exiting 19:12:21 (4068): No heartbeat from core client for 30 sec - exiting 19:12:22 (4068): No heartbeat from core client for 30 sec - exiting 19:12:23 (4068): No heartbeat from core client for 30 sec - exiting 19:12:24 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3036, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 00:27:09 (3676): No heartbeat from core client for 30 sec - exiting 00:27:11 (3676): No heartbeat from core client for 30 sec - exiting 00:27:12 (3676): No heartbeat from core client for 30 sec - exiting 00:27:13 (3676): No heartbeat from core client for 30 sec - exiting 00:27:14 (3676): No heartbeat from core client for 30 sec - exiting 00:27:15 (3676): No heartbeat from core client for 30 sec - exiting 00:27:16 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1016, iMonCtr=1 Model crash detected, will try to restart... 07:56:11 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:12 (4296): No heartbeat from core client for 30 sec - exiting 07:56:13 (4296): No heartbeat from core client for 30 sec - exiting 07:56:14 (4296): No heartbeat from core client for 30 sec - exiting 07:06:08 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:06 (3828): No heartbeat from core client for 30 sec - exiting 13:31:07 (3828): No heartbeat from core client for 30 sec - exiting 13:31:08 (3828): No heartbeat from core client for 30 sec - exiting 13:31:09 (3828): No heartbeat from core client for 30 sec - exiting 13:31:10 (3828): No heartbeat from core client for 30 sec - exiting 13:31:11 (3828): No heartbeat from core client for 30 sec - exiting 13:31:12 (3828): No heartbeat from core client for 30 sec - exiting 13:31:13 (3828): No heartbeat from core client for 30 sec - exiting 13:31:15 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:16 (3828): No heartbeat from core client for 30 sec - exiting 07:58:46 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:47 (4084): No heartbeat from core client for 30 sec - exiting 07:58:48 (4084): No heartbeat from core client for 30 sec - exiting 07:58:50 (4084): No heartbeat from core client for 30 sec - exiting 07:58:51 (4084): No heartbeat from core client for 30 sec - exiting 07:58:52 (4084): No heartbeat from core client for 30 sec - exiting 07:58:53 (4084): No heartbeat from core client for 30 sec - exiting 07:58:54 (4084): No heartbeat from core client for 30 sec - exiting 07:58:55 (4084): No heartbeat from core client for 30 sec - exiting 07:58:56 (4084): No heartbeat from core client for 30 sec - exiting 07:58:57 (4084): No heartbeat from core client for 30 sec - exiting 22:17:59 (3336): No heartbeat from core client for 30 sec - exiting 22:18:00 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:51 (2648): No heartbeat from core client for 30 sec - exiting 09:19:52 (2648): No heartbeat from core client for 30 sec - exiting 09:19:53 (2648): No heartbeat from core client for 30 sec - exiting 09:19:54 (2648): No heartbeat from core client for 30 sec - exiting 09:19:55 (2648): No heartbeat from core client for 30 sec - exiting 09:19:56 (2648): No heartbeat from core client for 30 sec - exiting 09:19:57 (2648): No heartbeat from core client for 30 sec - exiting 09:19:58 (2648): No heartbeat from core client for 30 sec - exiting 09:20:00 (2648): No heartbeat from core client for 30 sec - exiting 09:20:01 (2648): No heartbeat from core client for 30 sec - exiting 09:20:02 (2648): No heartbeat from core client for 30 sec - exiting 09:20:03 (2648): No heartbeat from core client for 30 sec - exiting 09:20:04 (2648): No heartbeat from core client for 30 sec - exiting 09:20:05 (2648): No heartbeat from core client for 30 sec - exiting 09:20:06 (2648): No heartbeat from core client for 30 sec - exiting 09:20:07 (2648): No heartbeat from core client for 30 sec - exiting 09:20:08 (2648): No heartbeat from core client for 30 sec - exiting 09:20:09 (2648): No heartbeat from core client for 30 sec - exiting 09:20:10 (2648): No heartbeat from core client for 30 sec - exiting 09:20:12 (2648): No heartbeat from core client for 30 sec - exiting 09:20:13 (2648): No heartbeat from core client for 30 sec - exiting 09:20:14 (2648): No heartbeat from core client for 30 sec - exiting 09:20:15 (2648): No heartbeat from core client for 30 sec - exiting 09:20:16 (2648): No heartbeat from core client for 30 sec - exiting 09:20:17 (2648): No heartbeat from core client for 30 sec - exiting 09:20:18 (2648): No heartbeat from core client for 30 sec - exiting 09:20:19 (2648): No heartbeat from core client for 30 sec - exiting 09:20:20 (2648): No heartbeat from core client for 30 sec - exiting 09:20:21 (2648): No heartbeat from core client for 30 sec - exiting 09:20:22 (2648): No heartbeat from core client for 30 sec - exiting 09:20:24 (2648): No heartbeat from core client for 30 sec - exiting 09:20:25 (2648): No heartbeat from core client for 30 sec - exiting 09:20:26 (2648): No heartbeat from core client for 30 sec - exiting 09:20:27 (2648): No heartbeat from core client for 30 sec - exiting 09:20:28 (2648): No heartbeat from core client for 30 sec - exiting 09:20:29 (2648): No heartbeat from core client for 30 sec - exiting 09:20:30 (2648): No heartbeat from core client for 30 sec - exiting 09:20:31 (2648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:32 (2648): No heartbeat from core client for 30 sec - exiting 00:01:39 (3812): No heartbeat from core client for 30 sec - exiting 00:01:40 (3812): No heartbeat from core client for 30 sec - exiting 00:01:41 (3812): No heartbeat from core client for 30 sec - exiting 00:01:42 (3812): No heartbeat from core client for 30 sec - exiting 00:01:43 (3812): No heartbeat from core client for 30 sec - exiting 00:01:44 (3812): No heartbeat from core client for 30 sec - exiting 00:01:45 (3812): No heartbeat from core client for 30 sec - exiting 00:01:47 (3812): No heartbeat from core client for 30 sec - exiting 00:01:48 (3812): No heartbeat from core client for 30 sec - exiting 00:01:49 (3812): No heartbeat from core client for 30 sec - exiting 00:01:50 (3812): No heartbeat from core client for 30 sec - exiting 00:01:51 (3812): No heartbeat from core client for 30 sec - exiting 00:01:52 (3812): No heartbeat from core client for 30 sec - exiting 00:01:53 (3812): No heartbeat from core client for 30 sec - exiting 00:01:54 (3812): No heartbeat from core client for 30 sec - exiting 00:01:55 (3812): No heartbeat from core client for 30 sec - exiting 00:01:56 (3812): No heartbeat from core client for 30 sec - exiting 00:01:58 (3812): No heartbeat from core client for 30 sec - exiting 00:01:59 (3812): No heartbeat from core client for 30 sec - exiting 00:02:00 (3812): No heartbeat from core client for 30 sec - exiting 00:02:01 (3812): No heartbeat from core client for 30 sec - exiting 00:02:02 (3812): No heartbeat from core client for 30 sec - exiting 00:02:03 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:39 (756): No heartbeat from core client for 30 sec - exiting 18:52:40 (756): No heartbeat from core client for 30 sec - exiting 18:52:41 (756): No heartbeat from core client for 30 sec - exiting 18:52:42 (756): No heartbeat from core client for 30 sec - exiting 18:52:44 (756): No heartbeat from core client for 30 sec - exiting 18:52:45 (756): No heartbeat from core client for 30 sec - exiting 18:52:46 (756): No heartbeat from core client for 30 sec - exiting 18:52:47 (756): No heartbeat from core client for 30 sec - exiting 18:52:48 (756): No heartbeat from core client for 30 sec - exiting 18:52:49 (756): No heartbeat from core client for 30 sec - exiting 18:52:50 (756): No heartbeat from core client for 30 sec - exiting 18:52:51 (756): No heartbeat from core client for 30 sec - exiting 18:52:52 (756): No heartbeat from core client for 30 sec - exiting 18:52:53 (756): No heartbeat from core client for 30 sec - exiting 18:52:54 (756): No heartbeat from core client for 30 sec - exiting 18:52:56 (756): No heartbeat from core client for 30 sec - exiting 18:52:57 (756): No heartbeat from core client for 30 sec - exiting 18:52:58 (756): No heartbeat from core client for 30 sec - exiting 18:52:59 (756): No heartbeat from core client for 30 sec - exiting 18:53:00 (756): No heartbeat from core client for 30 sec - exiting 18:53:01 (756): No heartbeat from core client for 30 sec - exiting 18:53:02 (756): No heartbeat from core client for 30 sec - exiting 18:53:03 (756): No heartbeat from core client for 30 sec - exiting 18:53:04 (756): No heartbeat from core client for 30 sec - exiting 18:53:05 (756): No heartbeat from core client for 30 sec - exiting 18:53:06 (756): No heartbeat from core client for 30 sec - exiting 18:53:08 (756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7723B84B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 20:30:37 (776): No heartbeat from core client for 30 sec - exiting 20:30:38 (776): No heartbeat from core client for 30 sec - exiting 20:30:39 (776): No heartbeat from core client for 30 sec - exiting 20:30:40 (776): No heartbeat from core client for 30 sec - exiting 20:30:41 (776): No heartbeat from core client for 30 sec - exiting 20:30:42 (776): No heartbeat from core client for 30 sec - exiting 20:30:43 (776): No heartbeat from core client for 30 sec - exiting 20:30:44 (776): No heartbeat from core client for 30 sec - exiting 20:30:46 (776): No heartbeat from core client for 30 sec - exiting 20:30:47 (776): No heartbeat from core client for 30 sec - exiting 20:30:48 (776): No heartbeat from core client for 30 sec - exiting 20:30:49 (776): No heartbeat from core client for 30 sec - exiting 20:30:50 (776): No heartbeat from core client for 30 sec - exiting 20:30:51 (776): No heartbeat from core client for 30 sec - exiting 20:30:52 (776): No heartbeat from core client for 30 sec - exiting 20:30:53 (776): No heartbeat from core client for 30 sec - exiting 20:30:54 (776): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77147188 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yf3h_1900_40_007352423/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Aug 2011 23:04:57	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	259,200	476,447	1.8381
11 Aug 2011 12:36:36	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	233,280	427,815	1.8339
09 Aug 2011 15:35:52	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	207,360	379,815	1.8317
07 Aug 2011 11:10:35	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	181,440	329,826	1.8178
01 Aug 2011 12:48:00	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	155,520	278,747	1.7924
28 Jul 2011 01:26:03	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	129,600	228,464	1.7628
25 Jul 2011 22:28:09	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	103,680	181,862	1.7541
25 Jul 2011 22:07:36	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	77,760	145,154	1.8667
25 Jul 2011 19:14:06	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	51,840	97,583	1.8824
25 Jul 2011 17:47:04	1065973	13108716	hadcm3n_yf3h_1900_40_007352423_1	25,920	48,306	1.8637