Task 14625026

Name	hadcm3n_101w_1980_40_007954742_4
Workunit	8109854
Created	3 May 2012, 16:57:15 UTC
Sent	3 May 2012, 16:57:31 UTC
Report deadline	3 Aug 2012, 0:24:42 UTC
Received	13 Jun 2012, 11:36:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1036359
Run time	15 days 22 hours 59 min 24 sec
CPU time	13 days 8 hours 14 min 25 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.57 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:24 (374): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:25 (374): No heartbeat from core client for 30 sec - exiting 21:33:26 (374): No heartbeat from core client for 30 sec - exiting 21:33:27 (374): No heartbeat from core client for 30 sec - exiting 21:33:28 (374): No heartbeat from core client for 30 sec - exiting 21:33:29 (374): No heartbeat from core client for 30 sec - exiting 21:33:30 (374): No heartbeat from core client for 30 sec - exiting 21:33:31 (374): No heartbeat from core client for 30 sec - exiting 21:33:32 (374): No heartbeat from core client for 30 sec - exiting 21:33:33 (374): No heartbeat from core client for 30 sec - exiting 21:33:34 (374): No heartbeat from core client for 30 sec - exiting 21:33:35 (374): No heartbeat from core client for 30 sec - exiting 21:33:36 (374): No heartbeat from core client for 30 sec - exiting 21:33:37 (374): No heartbeat from core client for 30 sec - exiting 21:33:38 (374): No heartbeat from core client for 30 sec - exiting 21:33:39 (374): No heartbeat from core client for 30 sec - exiting 21:33:40 (374): No heartbeat from core client for 30 sec - exiting 21:33:41 (374): No heartbeat from core client for 30 sec - exiting 21:33:42 (374): No heartbeat from core client for 30 sec - exiting 21:33:43 (374): No heartbeat from core client for 30 sec - exiting 21:33:44 (374): No heartbeat from core client for 30 sec - exiting 21:33:45 (374): No heartbeat from core client for 30 sec - exiting 21:33:46 (374): No heartbeat from core client for 30 sec - exiting 21:33:47 (374): No heartbeat from core client for 30 sec - exiting 21:33:48 (374): No heartbeat from core client for 30 sec - exiting 21:33:49 (374): No heartbeat from core client for 30 sec - exiting 21:33:50 (374): No heartbeat from core client for 30 sec - exiting 21:33:51 (374): No heartbeat from core client for 30 sec - exiting 21:33:52 (374): No heartbeat from core client for 30 sec - exiting 21:33:53 (374): No heartbeat from core client for 30 sec - exiting 21:33:54 (374): No heartbeat from core client for 30 sec - exiting 21:33:55 (374): No heartbeat from core client for 30 sec - exiting 21:33:56 (374): No heartbeat from core client for 30 sec - exiting 21:33:57 (374): No heartbeat from core client for 30 sec - exiting 21:33:58 (374): No heartbeat from core client for 30 sec - exiting 21:33:59 (374): No heartbeat from core client for 30 sec - exiting 21:34:01 (374): No heartbeat from core client for 30 sec - exiting 21:34:02 (374): No heartbeat from core client for 30 sec - exiting 21:34:03 (374): No heartbeat from core client for 30 sec - exiting 21:34:04 (374): No heartbeat from core client for 30 sec - exiting 21:34:05 (374): No heartbeat from core client for 30 sec - exiting 21:34:06 (374): No heartbeat from core client for 30 sec - exiting 21:34:07 (374): No heartbeat from core client for 30 sec - exiting 21:34:08 (374): No heartbeat from core client for 30 sec - exiting 21:34:09 (374): No heartbeat from core client for 30 sec - exiting 21:34:10 (374): No heartbeat from core client for 30 sec - exiting 21:40:07 (623): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:40:08 (623): No heartbeat from core client for 30 sec - exiting 21:40:09 (623): No heartbeat from core client for 30 sec - exiting 21:40:10 (623): No heartbeat from core client for 30 sec - exiting 21:40:11 (623): No heartbeat from core client for 30 sec - exiting 21:40:12 (623): No heartbeat from core client for 30 sec - exiting 21:40:13 (623): No heartbeat from core client for 30 sec - exiting 21:40:15 (623): No heartbeat from core client for 30 sec - exiting 21:40:16 (623): No heartbeat from core client for 30 sec - exiting 21:40:17 (623): No heartbeat from core client for 30 sec - exiting 21:40:18 (623): No heartbeat from core client for 30 sec - exiting 21:40:19 (623): No heartbeat from core client for 30 sec - exiting 21:40:20 (623): No heartbeat from core client for 30 sec - exiting 21:40:21 (623): No heartbeat from core client for 30 sec - exiting 21:40:22 (623): No heartbeat from core client for 30 sec - exiting 21:40:23 (623): No heartbeat from core client for 30 sec - exiting 21:40:24 (623): No heartbeat from core client for 30 sec - exiting 21:40:25 (623): No heartbeat from core client for 30 sec - exiting 21:40:26 (623): No heartbeat from core client for 30 sec - exiting 21:40:27 (623): No heartbeat from core client for 30 sec - exiting 21:40:28 (623): No heartbeat from core client for 30 sec - exiting 21:40:29 (623): No heartbeat from core client for 30 sec - exiting 21:40:30 (623): No heartbeat from core client for 30 sec - exiting 21:40:31 (623): No heartbeat from core client for 30 sec - exiting 21:40:32 (623): No heartbeat from core client for 30 sec - exiting 21:40:33 (623): No heartbeat from core client for 30 sec - exiting 21:40:34 (623): No heartbeat from core client for 30 sec - exiting 21:40:35 (623): No heartbeat from core client for 30 sec - exiting 21:40:36 (623): No heartbeat from core client for 30 sec - exiting 21:40:37 (623): No heartbeat from core client for 30 sec - exiting 21:40:38 (623): No heartbeat from core client for 30 sec - exiting 21:40:39 (623): No heartbeat from core client for 30 sec - exiting 21:40:40 (623): No heartbeat from core client for 30 sec - exiting 21:40:41 (623): No heartbeat from core client for 30 sec - exiting 21:40:42 (623): No heartbeat from core client for 30 sec - exiting 21:40:43 (623): No heartbeat from core client for 30 sec - exiting 21:40:44 (623): No heartbeat from core client for 30 sec - exiting 21:40:45 (623): No heartbeat from core client for 30 sec - exiting 21:40:46 (623): No heartbeat from core client for 30 sec - exiting 21:40:47 (623): No heartbeat from core client for 30 sec - exiting 21:40:48 (623): No heartbeat from core client for 30 sec - exiting 21:40:49 (623): No heartbeat from core client for 30 sec - exiting 21:40:50 (623): No heartbeat from core client for 30 sec - exiting 21:40:51 (623): No heartbeat from core client for 30 sec - exiting 21:40:52 (623): No heartbeat from core client for 30 sec - exiting 21:40:53 (623): No heartbeat from core client for 30 sec - exiting 21:40:54 (623): No heartbeat from core client for 30 sec - exiting 21:40:55 (623): No heartbeat from core client for 30 sec - exiting 21:40:56 (623): No heartbeat from core client for 30 sec - exiting 21:40:57 (623): No heartbeat from core client for 30 sec - exiting 21:40:58 (623): No heartbeat from core client for 30 sec - exiting 21:40:59 (623): No heartbeat from core client for 30 sec - exiting 21:41:00 (623): No heartbeat from core client for 30 sec - exiting 21:41:01 (623): No heartbeat from core client for 30 sec - exiting 21:41:02 (623): No heartbeat from core client for 30 sec - exiting 21:41:03 (623): No heartbeat from core client for 30 sec - exiting 21:41:04 (623): No heartbeat from core client for 30 sec - exiting 21:41:05 (623): No heartbeat from core client for 30 sec - exiting 21:41:06 (623): No heartbeat from core client for 30 sec - exiting 21:41:07 (623): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12892,0x3cc2c0) malloc: * error for object 0x43be600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(22172,0x3cc2c0) malloc: * error for object 0x23bc204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(22172,0x3cc2c0) malloc: * error for object 0x23bc200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:40:25 (1966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:19 (3617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:19 (7691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:19:01 (11772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:51 (14299): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:12:42 (2652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:34:13 (4598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:43:45 (2488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:46 (2488): No heartbeat from core client for 30 sec - exiting 18:43:47 (2488): No heartbeat from core client for 30 sec - exiting 18:43:48 (2488): No heartbeat from core client for 30 sec - exiting 18:43:49 (2488): No heartbeat from core client for 30 sec - exiting 18:43:50 (2488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:52:23 (41587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:15:11 (47171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:16 (49719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:13:00 (349): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:11 (430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Jun 2012 14:17:14	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	1,010,880	1,128,712	1.1166
11 Jun 2012 12:33:56	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	984,960	1,101,702	1.1185
09 Jun 2012 20:24:36	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	959,040	1,072,846	1.1187
09 Jun 2012 07:20:20	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	933,120	1,043,899	1.1187
07 Jun 2012 00:30:00	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	907,200	1,015,485	1.1194
05 Jun 2012 21:42:24	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	881,280	986,995	1.1200
04 Jun 2012 21:31:44	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	855,360	958,385	1.1204
03 Jun 2012 04:21:22	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	829,440	929,719	1.1209
02 Jun 2012 18:45:21	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	803,520	900,656	1.1209
30 May 2012 21:33:55	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	777,600	871,504	1.1208
30 May 2012 09:45:26	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	751,680	842,914	1.1214
28 May 2012 16:50:40	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	725,760	814,590	1.1224
27 May 2012 11:34:54	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	699,840	786,119	1.1233
25 May 2012 21:35:05	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	673,920	757,806	1.1245
25 May 2012 09:59:06	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	648,000	729,456	1.1257
25 May 2012 00:21:34	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	622,080	699,887	1.1251
24 May 2012 14:53:05	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	596,160	671,190	1.1259
24 May 2012 04:34:13	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	570,240	642,141	1.1261
23 May 2012 20:01:46	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	544,320	613,584	1.1272
21 May 2012 21:23:01	1036359	14625026	hadcm3n_101w_1980_40_007954742_4	518,400	584,542	1.1276