Task 15896176

Name	hadcm3n_3b8w_1980_40_008283708_3
Workunit	8434843
Created	18 Jul 2013, 15:52:59 UTC
Sent	18 Jul 2013, 16:01:05 UTC
Report deadline	17 Oct 2013, 23:28:16 UTC
Received	15 Oct 2013, 14:17:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1146560
Run time	16 days 14 hours 11 min 8 sec
CPU time	14 days 2 hours 52 min 35 sec
Validate state	Invalid
Credit	9,020.16
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 07:58:58 (194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:59 (194): No heartbeat from core client for 30 sec - exiting 07:59:00 (194): No heartbeat from core client for 30 sec - exiting 07:59:01 (194): No heartbeat from core client for 30 sec - exiting 07:59:02 (194): No heartbeat from core client for 30 sec - exiting 07:59:04 (194): No heartbeat from core client for 30 sec - exiting 07:59:05 (194): No heartbeat from core client for 30 sec - exiting 07:59:06 (194): No heartbeat from core client for 30 sec - exiting 07:59:07 (194): No heartbeat from core client for 30 sec - exiting 07:59:10 (194): No heartbeat from core client for 30 sec - exiting 07:59:11 (194): No heartbeat from core client for 30 sec - exiting 07:59:12 (194): No heartbeat from core client for 30 sec - exiting 07:59:13 (194): No heartbeat from core client for 30 sec - exiting 07:59:14 (194): No heartbeat from core client for 30 sec - exiting 07:59:15 (194): No heartbeat from core client for 30 sec - exiting 07:59:16 (194): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:15:28 (180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1022204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1022200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1022204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1022200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x821200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(196,0xa0189540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:43 (193): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:44 (193): No heartbeat from core client for 30 sec - exiting 08:16:45 (193): No heartbeat from core client for 30 sec - exiting 08:16:46 (193): No heartbeat from core client for 30 sec - exiting 08:16:47 (193): No heartbeat from core client for 30 sec - exiting 08:16:48 (193): No heartbeat from core client for 30 sec - exiting 08:16:49 (193): No heartbeat from core client for 30 sec - exiting 08:16:50 (193): No heartbeat from core client for 30 sec - exiting 08:16:51 (193): No heartbeat from core client for 30 sec - exiting 08:16:53 (193): No heartbeat from core client for 30 sec - exiting 08:16:54 (193): No heartbeat from core client for 30 sec - exiting 08:16:55 (193): No heartbeat from core client for 30 sec - exiting 08:16:56 (193): No heartbeat from core client for 30 sec - exiting 08:16:57 (193): No heartbeat from core client for 30 sec - exiting 08:16:58 (193): No heartbeat from core client for 30 sec - exiting 08:16:59 (193): No heartbeat from core client for 30 sec - exiting 08:17:00 (193): No heartbeat from core client for 30 sec - exiting 08:17:01 (193): No heartbeat from core client for 30 sec - exiting 08:17:02 (193): No heartbeat from core client for 30 sec - exiting 08:17:03 (193): No heartbeat from core client for 30 sec - exiting 08:17:04 (193): No heartbeat from core client for 30 sec - exiting 08:17:05 (193): No heartbeat from core client for 30 sec - exiting 08:17:06 (193): No heartbeat from core client for 30 sec - exiting 08:17:07 (193): No heartbeat from core client for 30 sec - exiting 08:17:08 (193): No heartbeat from core client for 30 sec - exiting 08:17:10 (193): No heartbeat from core client for 30 sec - exiting 08:17:11 (193): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1069604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1069600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x104e004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1069604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1069600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x842600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x827004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x104e004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1012c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x1012c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x810800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(182,0xa0189540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 08:03:46 (175): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:48 (175): No heartbeat from core client for 30 sec - exiting 08:03:50 (175): No heartbeat from core client for 30 sec - exiting 08:03:52 (175): No heartbeat from core client for 30 sec - exiting 08:03:54 (175): No heartbeat from core client for 30 sec - exiting 08:03:56 (175): No heartbeat from core client for 30 sec - exiting 08:03:57 (175): No heartbeat from core client for 30 sec - exiting 08:04:02 (175): No heartbeat from core client for 30 sec - exiting 08:04:04 (175): No heartbeat from core client for 30 sec - exiting 08:04:09 (175): No heartbeat from core client for 30 sec - exiting 08:04:10 (175): No heartbeat from core client for 30 sec - exiting 08:04:11 (175): No heartbeat from core client for 30 sec - exiting 08:04:13 (175): No heartbeat from core client for 30 sec - exiting 08:04:14 (175): No heartbeat from core client for 30 sec - exiting 08:04:15 (175): No heartbeat from core client for 30 sec - exiting 08:04:16 (175): No heartbeat from core client for 30 sec - exiting 08:04:17 (175): No heartbeat from core client for 30 sec - exiting 08:04:18 (175): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:16:56 (187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:16:57 (187): No heartbeat from core client for 30 sec - exiting 09:16:58 (187): No heartbeat from core client for 30 sec - exiting 09:17:00 (187): No heartbeat from core client for 30 sec - exiting 09:17:01 (187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:26:10 (187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:11 (187): No heartbeat from core client for 30 sec - exiting 07:26:12 (187): No heartbeat from core client for 30 sec - exiting 07:26:13 (187): No heartbeat from core client for 30 sec - exiting 07:26:14 (187): No heartbeat from core client for 30 sec - exiting 07:26:18 (187): No heartbeat from core client for 30 sec - exiting 07:26:19 (187): No heartbeat from core client for 30 sec - exiting 07:26:20 (187): No heartbeat from core client for 30 sec - exiting 07:26:21 (187): No heartbeat from core client for 30 sec - exiting 07:26:22 (187): No heartbeat from core client for 30 sec - exiting 07:26:23 (187): No heartbeat from core client for 30 sec - exiting 07:26:26 (187): No heartbeat from core client for 30 sec - exiting 07:26:28 (187): No heartbeat from core client for 30 sec - exiting 07:26:30 (187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Oct 2013 19:17:43	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	751,680	1,206,172	1.6046
08 Oct 2013 18:10:20	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	725,760	1,166,265	1.6070
05 Oct 2013 11:45:00	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	699,840	1,123,671	1.6056
04 Oct 2013 23:09:34	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	673,920	1,081,160	1.6043
03 Oct 2013 18:38:30	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	648,000	1,038,754	1.6030
30 Sep 2013 19:44:17	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	622,080	996,354	1.6016
25 Sep 2013 22:38:06	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	596,160	954,043	1.6003
25 Sep 2013 09:22:06	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	570,240	911,209	1.5979
19 Sep 2013 22:35:27	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	544,320	868,861	1.5962
18 Sep 2013 16:41:32	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	518,400	826,554	1.5944
16 Sep 2013 17:44:20	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	492,480	784,432	1.5928
11 Sep 2013 18:08:33	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	466,560	742,827	1.5921
09 Sep 2013 18:08:15	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	440,640	701,013	1.5909
04 Sep 2013 20:40:17	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	414,720	658,561	1.5880
03 Sep 2013 15:33:17	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	388,800	616,237	1.5850
28 Aug 2013 19:56:20	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	362,880	573,739	1.5811
26 Aug 2013 21:52:28	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	336,960	531,309	1.5768
26 Aug 2013 15:05:42	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	311,040	489,009	1.5722
20 Aug 2013 18:26:07	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	285,120	446,742	1.5669
15 Aug 2013 19:54:59	1146560	15896176	hadcm3n_3b8w_1980_40_008283708_3	259,200	404,347	1.5600