Task 19418258

Name	hadcm3n_xde1_198012_480_374_010412714_0
Workunit	10412714
Created	21 Mar 2016, 15:15:57 UTC
Sent	24 Mar 2016, 3:38:23 UTC
Report deadline	6 Mar 2017, 8:58:23 UTC
Received	16 Apr 2016, 8:37:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1375395
Run time	11 days 17 hours 27 min 2 sec
CPU time	11 days 2 hours 38 min 1 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.71 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:46:09 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:47:41 (8955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:08:00 (13366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:19:24 (18468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:59:53 (25457): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:27:06 (33547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:27:42 (41179): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:01:36 (49927): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:47 (55838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:14 (65294): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:38:14 (76564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:30 (83520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:56:28 (92780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:11 (99548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:09:32 (10952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:28 (17910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:38:28 (22284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:18:48 (58028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:16:53 (78462): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:19:43 (84119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:25:30 (92934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:44:09 (99701): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:23 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:11:34 (11939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:54 (19108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:44 (26881): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:49:22 (34886): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:24:57 (42237): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:23:31 (50289): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:39:24 (56738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:08:57 (61717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:56:16 (67128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:06:16 (79788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:16:19 (84487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:53:04 (9571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:00:18 (24286): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:03:41 (28849): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:16:45 (44824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:46:59 (53625): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:22:53 (63472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:19 (71327): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:18:37 (97224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:49:24 (2090): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:51 (7548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:25:06 (49415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:41:28 (85587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:28:35 (91267): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:07 (97350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:53:07 (24641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:58:07 (55302): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:31:07 (78977): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:34:05 (89752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:47 (94200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:35:50 (1260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:42:36 (48405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:04:01 (53763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:21:51 (72385): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:39:56 (16211): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:39:27 (21643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:11:41 (30464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:43:36 (38294): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(43866,0xa43df000) malloc: * error for object 0x2839204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation 08:27:26 (43866): No heartbeat from core client for 30 sec - exiting 08:27:27 (43866): No heartbeat from core client for 30 sec - exiting 08:27:28 (43866): No heartbeat from core client for 30 sec - exiting 08:27:29 (43866): No heartbeat from core client for 30 sec - exiting 08:27:30 (43866): No heartbeat from core client for 30 sec - exiting 08:27:31 (43866): No heartbeat from core client for 30 sec - exiting 08:27:32 (43866): No heartbeat from core client for 30 sec - exiting 08:27:33 (43866): No heartbeat from core client for 30 sec - exiting 08:27:34 (43866): No heartbeat from core client for 30 sec - exiting 08:27:35 (43866): No heartbeat from core client for 30 sec - exiting 08:27:36 (43866): No heartbeat from core client for 30 sec - exiting 08:27:37 (43866): No heartbeat from core client for 30 sec - exiting 08:27:38 (43866): No heartbeat from core client for 30 sec - exiting 08:27:39 (43866): No heartbeat from core client for 30 sec - exiting 08:27:40 (43866): No heartbeat from core client for 30 sec - exiting 08:27:42 (43866): No heartbeat from core client for 30 sec - exiting 08:27:43 (43866): No heartbeat from core client for 30 sec - exiting 08:27:44 (43866): No heartbeat from core client for 30 sec - exiting 08:27:45 (43866): No heartbeat from core client for 30 sec - exiting 08:27:46 (43866): No heartbeat from core client for 30 sec - exiting 08:27:47 (43866): No heartbeat from core client for 30 sec - exiting 08:27:48 (43866): No heartbeat from core client for 30 sec - exiting 08:27:49 (43866): No heartbeat from core client for 30 sec - exiting 08:27:50 (43866): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57567,0xa43df000) malloc: * error for object 0x839e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x282a004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x282a000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x2003004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x2003000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x2000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x2000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57811,0xa43df000) malloc: * error for object 0x2000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation CPDN Monitor - Quit request from BOINC... 10:57:59 (59621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:57:22 (68896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:12:29 (77969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:19:17 (87300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:37:58 (13663): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:23:32 (20940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:35:40 (31447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:34:22 (38575): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:44:28 (70901): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:47:43 (77323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:41:26 (82934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:03:16 (99314): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:30:20 (45185): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:31:49 (54006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:43:21 (58450): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:28 (86816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:17:35 (93321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:26:08 (98321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:06:46 (3367): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:23:04 (20409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:56:54 (30084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:00:20 (37774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:26:27 (42317): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:01:21 (47678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:43 (59999): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:39:02 (65782): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:55:12 (74669): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:22:46 (95392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:26:02 (2411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:06:09 (6931): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:09:25 (12812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:28:50 (31490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:59:11 (36394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:29:31 (68350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:10 (73405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:05:23 (82178): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:05:06 (97672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:15:47 (4547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:43 (9954): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:23:16 (33415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:34:47 (45169): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:58:35 (49980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:29:13 (2726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:25 (19195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:45:55 (28872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:56:47 (35324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:02:22 (48031): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:11:49 (11284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:11:54 (10843): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:42:41 (20468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:57:13 (27789): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(58712,0xa3cfe000) malloc: * error for object 0x2000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58712,0xa3cfe000) malloc: * error for object 0x2001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel x86-64h Haswell (32-bit executable) 19:04:59 (58712): No heartbeat from core client for 30 sec - exiting 19:05:00 (58712): No heartbeat from core client for 30 sec - exiting 19:05:01 (58712): No heartbeat from core client for 30 sec - exiting 19:05:02 (58712): No heartbeat from core client for 30 sec - exiting 19:05:03 (58712): No heartbeat from core client for 30 sec - exiting 19:05:04 (58712): No heartbeat from core client for 30 sec - exiting 19:05:05 (58712): No heartbeat from core client for 30 sec - exiting 19:05:06 (58712): No heartbeat from core client for 30 sec - exiting 19:05:07 (58712): No heartbeat from core client for 30 sec - exiting 19:05:08 (58712): No heartbeat from core client for 30 sec - exiting 19:05:09 (58712): No heartbeat from core client for 30 sec - exiting 19:05:10 (58712): No heartbeat from core client for 30 sec - exiting 19:05:12 (58712): No heartbeat from core client for 30 sec - exiting 19:05:13 (58712): No heartbeat from core client for 30 sec - exiting 19:05:14 (58712): No heartbeat from core client for 30 sec - exiting 19:05:15 (58712): No heartbeat from core client for 30 sec - exiting 19:05:16 (58712): No heartbeat from core client for 30 sec - exiting 19:05:17 (58712): No heartbeat from core client for 30 sec - exiting 20:06:36 (58712): No heartbeat from core client for 30 sec - exiting 20:06:37 (58712): No heartbeat from core client for 30 sec - exiting 20:06:38 (58712): No heartbeat from core client for 30 sec - exiting 20:06:39 (58712): No heartbeat from core client for 30 sec - exiting 20:06:40 (58712): No heartbeat from core client for 30 sec - exiting 20:06:41 (58712): No heartbeat from core client for 30 sec - exiting 20:06:42 (58712): No heartbeat from core client for 30 sec - exiting 20:06:43 (58712): No heartbeat from core client for 30 sec - exiting 20:06:44 (58712): No heartbeat from core client for 30 sec - exiting 20:06:45 (58712): No heartbeat from core client for 30 sec - exiting 20:06:46 (58712): No heartbeat from core client for 30 sec - exiting 20:06:47 (58712): No heartbeat from core client for 30 sec - exiting 20:06:48 (58712): No heartbeat from core client for 30 sec - exiting 20:06:49 (58712): No heartbeat from core client for 30 sec - exiting 20:06:50 (58712): No heartbeat from core client for 30 sec - exiting 20:06:51 (58712): No heartbeat from core client for 30 sec - exiting 20:06:52 (58712): No heartbeat from core client for 30 sec - exiting 20:06:54 (58712): No heartbeat from core client for 30 sec - exiting 20:06:55 (58712): No heartbeat from core client for 30 sec - exiting 20:06:56 (58712): No heartbeat from core client for 30 sec - exiting 20:06:57 (58712): No heartbeat from core client for 30 sec - exiting 20:06:58 (58712): No heartbeat from core client for 30 sec - exiting 20:06:59 (58712): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(84776,0xa3cfe000) malloc: * error for object 0x1002000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel x86-64h Haswell (32-bit executable) System version: Macintosh OS 10.11.4 build 15E65 Sat Apr 16 00:37:20 2016 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd38 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/6/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x908a9000 - 0x908a9fff /usr/lib/system/libunc.dylib 0x93117000 - 0x9316afff /usr/lib/libstdc++.6.dylib 0x931d6000 - 0x93244fff /usr/lib/system/libsystem_network.dylib 0x93b5e000 - 0x93b64fff /usr/lib/system/libunwind.dylib 0x9425b000 - 0x94272fff /usr/lib/system/libsystem_coretls.dylib 0x9429c000 - 0x942a5fff /usr/lib/system/libcopyfile.dylib 0x94556000 - 0x9456ffff /usr/lib/system/libsystem_asl.dylib 0x94635000 - 0x9465dfff /usr/lib/system/libxpc.dylib 0x95408000 - 0x9549dfff /usr/lib/system/libsystem_c.dylib 0x95557000 - 0x9555afff /usr/lib/system/libdyld.dylib 0x967f4000 - 0x967f4fff /usr/lib/system/libsystem_blocks.dylib 0x9749a000 - 0x9749cfff /usr/lib/system/libquarantine.dylib 0x97f53000 - 0x97f58fff /usr/lib/system/libcompiler_rt.dylib 0x97fc4000 - 0x97fc4fff /usr/lib/system/libkeymgr.dylib 0x97fc5000 - 0x97fcbfff /usr/lib/system/libsystem_platform.dylib 0x989e0000 - 0x989eafff /usr/lib/system/libsystem_notify.dylib 0x98ced000 - 0x98cf5fff /usr/lib/system/libsystem_pthread.dylib 0x99489000 - 0x9948bfff /usr/lib/system/libsystem_configuration.dylib 0x9948c000 - 0x994f3fff /usr/lib/system/libcorecrypto.dylib 0x994f4000 - 0x9951efff /usr/lib/system/libdispatch.dylib 0x99b74000 - 0x99b7ffff /usr/lib/system/libcommonCrypto.dylib 0x99db4000 - 0x99db5fff /usr/lib/libSystem.B.dylib 0x99f34000 - 0x99f3dfff /usr/lib/system/libsystem_networkextension.dylib 0x9a015000 - 0x9a015fff /usr/lib/system/liblaunch.dylib 0x9b25d000 - 0x9b287fff /usr/lib/system/libsystem_info.dylib 0x9b28e000 - 0x9b296fff /usr/lib/system/libsystem_dnssd.dylib 0x9b8d9000 - 0x9b8defff /usr/lib/system/libmacho.dylib 0x9b928000 - 0x9b92bfff /usr/lib/system/libsystem_sandbox.dylib 0x9c06e000 - 0x9c070fff /usr/lib/system/libsystem_coreservices.dylib 0x9c080000 - 0x9c0a1fff /usr/lib/system/libsystem_kernel.dylib 0x9c317000 - 0x9c318fff /usr/lib/system/libremovefile.dylib 0x9c371000 - 0x9c373fff /usr/lib/system/libsystem_secinit.dylib 0x9c438000 - 0x9c449fff /usr/lib/system/libsystem_trace.dylib 0x9c472000 - 0x9c495fff /usr/lib/libc++abi.dylib 0x9c496000 - 0x9c4affff /usr/lib/system/libsystem_malloc.dylib 0x9c5f6000 - 0x9c604fff /usr/lib/libz.1.dylib 0x9d0ad000 - 0x9d0e0fff /usr/lib/system/libsystem_m.dylib 0x9d23f000 - 0x9d243fff /usr/lib/system/libcache.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Apr 2016 01:29:08	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	518,400	938,994	1.8113
15 Apr 2016 12:15:32	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	492,480	890,597	1.8084
15 Apr 2016 12:12:52	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	466,560	842,830	1.8065
15 Apr 2016 12:08:44	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	440,640	795,904	1.8062
10 Apr 2016 09:29:01	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	414,720	744,253	1.7946
09 Apr 2016 04:48:42	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	388,800	693,670	1.7841
07 Apr 2016 11:32:42	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	362,880	641,340	1.7674
06 Apr 2016 09:11:01	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	336,960	586,345	1.7401
03 Apr 2016 17:53:22	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	311,040	540,575	1.7380
03 Apr 2016 03:10:25	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	285,120	493,619	1.7313
02 Apr 2016 13:11:02	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	259,200	449,457	1.7340
01 Apr 2016 11:01:55	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	233,280	401,794	1.7224
30 Mar 2016 08:18:15	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	207,360	356,168	1.7176
28 Mar 2016 15:58:12	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	181,440	312,276	1.7211
27 Mar 2016 20:55:03	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	155,520	275,462	1.7712
27 Mar 2016 04:51:25	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	129,600	225,679	1.7414
26 Mar 2016 14:23:17	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	103,680	176,286	1.7003
26 Mar 2016 01:05:06	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	77,760	134,481	1.7294
25 Mar 2016 12:19:21	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	51,840	93,804	1.8095
24 Mar 2016 22:11:53	1375395	19418258	hadcm3n_xde1_198012_480_374_010412714_0	25,920	46,592	1.7975