Task 15992270

Name	hadcm3n_80cf_1980_40_008458386_0
Workunit	8609242
Created	30 Aug 2013, 20:13:56 UTC
Sent	8 Sep 2013, 9:51:42 UTC
Report deadline	8 Dec 2013, 17:18:53 UTC
Received	26 Jul 2014, 12:04:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1022874
Run time	12 days 17 hours 36 min 52 sec
CPU time	8 days 13 hours 47 min 22 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 14:05:22 (30113): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:57:08 (2273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:10 (2273): No heartbeat from core client for 30 sec - exiting 13:57:11 (2273): No heartbeat from core client for 30 sec - exiting 13:57:12 (2273): No heartbeat from core client for 30 sec - exiting 13:57:13 (2273): No heartbeat from core client for 30 sec - exiting 13:57:14 (2273): No heartbeat from core client for 30 sec - exiting 13:57:15 (2273): No heartbeat from core client for 30 sec - exiting 13:57:16 (2273): No heartbeat from core client for 30 sec - exiting 13:57:17 (2273): No heartbeat from core client for 30 sec - exiting 13:57:18 (2273): No heartbeat from core client for 30 sec - exiting 13:57:19 (2273): No heartbeat from core client for 30 sec - exiting 13:57:20 (2273): No heartbeat from core client for 30 sec - exiting 13:57:21 (2273): No heartbeat from core client for 30 sec - exiting 13:57:22 (2273): No heartbeat from core client for 30 sec - exiting 13:57:23 (2273): No heartbeat from core client for 30 sec - exiting 13:57:24 (2273): No heartbeat from core client for 30 sec - exiting 13:57:25 (2273): No heartbeat from core client for 30 sec - exiting 13:57:26 (2273): No heartbeat from core client for 30 sec - exiting 13:57:27 (2273): No heartbeat from core client for 30 sec - exiting 13:57:28 (2273): No heartbeat from core client for 30 sec - exiting 13:57:29 (2273): No heartbeat from core client for 30 sec - exiting 13:57:30 (2273): No heartbeat from core client for 30 sec - exiting 13:57:31 (2273): No heartbeat from core client for 30 sec - exiting 13:57:32 (2273): No heartbeat from core client for 30 sec - exiting 13:57:33 (2273): No heartbeat from core client for 30 sec - exiting 13:57:34 (2273): No heartbeat from core client for 30 sec - exiting 13:57:35 (2273): No heartbeat from core client for 30 sec - exiting 13:57:36 (2273): No heartbeat from core client for 30 sec - exiting 13:57:37 (2273): No heartbeat from core client for 30 sec - exiting 13:57:38 (2273): No heartbeat from core client for 30 sec - exiting 13:57:39 (2273): No heartbeat from core client for 30 sec - exiting 13:57:40 (2273): No heartbeat from core client for 30 sec - exiting 13:57:41 (2273): No heartbeat from core client for 30 sec - exiting 13:57:42 (2273): No heartbeat from core client for 30 sec - exiting 13:57:43 (2273): No heartbeat from core client for 30 sec - exiting 13:57:44 (2273): No heartbeat from core client for 30 sec - exiting 13:57:45 (2273): No heartbeat from core client for 30 sec - exiting 13:57:46 (2273): No heartbeat from core client for 30 sec - exiting 13:57:47 (2273): No heartbeat from core client for 30 sec - exiting 13:57:48 (2273): No heartbeat from core client for 30 sec - exiting 13:57:49 (2273): No heartbeat from core client for 30 sec - exiting 13:57:50 (2273): No heartbeat from core client for 30 sec - exiting 13:18:00 (2660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:53:06 (279): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:08 (279): No heartbeat from core client for 30 sec - exiting 07:45:24 (2993): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:41:02 (63812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:35 (39660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:36 (39660): No heartbeat from core client for 30 sec - exiting 18:45:37 (39660): No heartbeat from core client for 30 sec - exiting 18:45:38 (39660): No heartbeat from core client for 30 sec - exiting 18:45:39 (39660): No heartbeat from core client for 30 sec - exiting 18:45:40 (39660): No heartbeat from core client for 30 sec - exiting 18:45:41 (39660): No heartbeat from core client for 30 sec - exiting 18:45:42 (39660): No heartbeat from core client for 30 sec - exiting 18:45:43 (39660): No heartbeat from core client for 30 sec - exiting 18:45:44 (39660): No heartbeat from core client for 30 sec - exiting 18:45:45 (39660): No heartbeat from core client for 30 sec - exiting 18:45:46 (39660): No heartbeat from core client for 30 sec - exiting 18:45:47 (39660): No heartbeat from core client for 30 sec - exiting 18:45:48 (39660): No heartbeat from core client for 30 sec - exiting 18:45:49 (39660): No heartbeat from core client for 30 sec - exiting 18:45:50 (39660): No heartbeat from core client for 30 sec - exiting 18:45:51 (39660): No heartbeat from core client for 30 sec - exiting 18:45:52 (39660): No heartbeat from core client for 30 sec - exiting 18:45:53 (39660): No heartbeat from core client for 30 sec - exiting 18:45:54 (39660): No heartbeat from core client for 30 sec - exiting 18:45:55 (39660): No heartbeat from core client for 30 sec - exiting 18:50:32 (39694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:34 (39712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1802800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(39750,0xa0762540) malloc: * error for object 0x1800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 09:57:05 (39750): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:10 (39750): No heartbeat from core client for 30 sec - exiting 09:57:11 (39750): No heartbeat from core client for 30 sec - exiting 09:57:12 (39750): No heartbeat from core client for 30 sec - exiting 09:57:13 (39750): No heartbeat from core client for 30 sec - exiting 09:57:14 (39750): No heartbeat from core client for 30 sec - exiting 09:57:15 (39750): No heartbeat from core client for 30 sec - exiting 09:57:16 (39750): No heartbeat from core client for 30 sec - exiting 09:57:17 (39750): No heartbeat from core client for 30 sec - exiting 09:57:18 (39750): No heartbeat from core client for 30 sec - exiting 09:57:19 (39750): No heartbeat from core client for 30 sec - exiting 09:57:20 (39750): No heartbeat from core client for 30 sec - exiting 09:57:21 (39750): No heartbeat from core client for 30 sec - exiting 09:57:22 (39750): No heartbeat from core client for 30 sec - exiting 09:57:23 (39750): No heartbeat from core client for 30 sec - exiting 09:57:24 (39750): No heartbeat from core client for 30 sec - exiting 09:57:25 (39750): No heartbeat from core client for 30 sec - exiting 09:57:26 (39750): No heartbeat from core client for 30 sec - exiting 09:57:27 (39750): No heartbeat from core client for 30 sec - exiting 09:57:28 (39750): No heartbeat from core client for 30 sec - exiting 09:57:29 (39750): No heartbeat from core client for 30 sec - exiting 09:57:30 (39750): No heartbeat from core client for 30 sec - exiting 09:57:31 (39750): No heartbeat from core client for 30 sec - exiting 09:57:32 (39750): No heartbeat from core client for 30 sec - exiting 09:57:33 (39750): No heartbeat from core client for 30 sec - exiting 09:57:34 (39750): No heartbeat from core client for 30 sec - exiting 09:57:35 (39750): No heartbeat from core client for 30 sec - exiting 09:57:36 (39750): No heartbeat from core client for 30 sec - exiting 09:57:37 (39750): No heartbeat from core client for 30 sec - exiting 09:57:38 (39750): No heartbeat from core client for 30 sec - exiting 09:57:39 (39750): No heartbeat from core client for 30 sec - exiting 09:57:40 (39750): No heartbeat from core client for 30 sec - exiting 09:57:41 (39750): No heartbeat from core client for 30 sec - exiting 09:57:42 (39750): No heartbeat from core client for 30 sec - exiting 09:57:43 (39750): No heartbeat from core client for 30 sec - exiting 09:57:44 (39750): No heartbeat from core client for 30 sec - exiting 09:57:45 (39750): No heartbeat from core client for 30 sec - exiting 09:57:46 (39750): No heartbeat from core client for 30 sec - exiting 09:57:47 (39750): No heartbeat from core client for 30 sec - exiting 09:57:48 (39750): No heartbeat from core client for 30 sec - exiting 09:57:49 (39750): No heartbeat from core client for 30 sec - exiting 09:57:50 (39750): No heartbeat from core client for 30 sec - exiting 09:57:51 (39750): No heartbeat from core client for 30 sec - exiting 09:57:52 (39750): No heartbeat from core client for 30 sec - exiting 09:57:54 (39750): No heartbeat from core client for 30 sec - exiting 09:57:55 (39750): No heartbeat from core client for 30 sec - exiting 09:57:56 (39750): No heartbeat from core client for 30 sec - exiting 09:57:57 (39750): No heartbeat from core client for 30 sec - exiting 09:57:58 (39750): No heartbeat from core client for 30 sec - exiting 09:57:59 (39750): No heartbeat from core client for 30 sec - exiting 09:58:00 (39750): No heartbeat from core client for 30 sec - exiting 09:58:01 (39750): No heartbeat from core client for 30 sec - exiting 09:58:02 (39750): No heartbeat from core client for 30 sec - exiting 09:58:03 (39750): No heartbeat from core client for 30 sec - exiting 09:58:04 (39750): No heartbeat from core client for 30 sec - exiting 09:58:05 (39750): No heartbeat from core client for 30 sec - exiting 09:58:06 (39750): No heartbeat from core client for 30 sec - exiting 09:58:07 (39750): No heartbeat from core client for 30 sec - exiting 09:58:08 (39750): No heartbeat from core client for 30 sec - exiting 09:58:09 (39750): No heartbeat from core client for 30 sec - exiting 09:58:10 (39750): No heartbeat from core client for 30 sec - exiting 09:58:11 (39750): No heartbeat from core client for 30 sec - exiting 09:58:12 (39750): No heartbeat from core client for 30 sec - exiting 09:58:13 (39750): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:19:19 (45798): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:26:53 (62937): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of atmos prognostic fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 May 2014 19:19:58	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	466,560	724,792	1.5535
15 May 2014 02:42:17	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	440,640	684,346	1.5531
10 May 2014 23:49:21	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	414,720	645,458	1.5564
03 May 2014 15:23:46	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	388,800	604,881	1.5558
18 Apr 2014 13:48:58	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	362,880	563,869	1.5539
14 Apr 2014 03:19:30	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	336,960	523,560	1.5538
11 Apr 2014 01:11:49	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	311,040	483,239	1.5536
08 Apr 2014 05:10:18	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	285,120	442,847	1.5532
27 Feb 2014 01:08:57	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	259,200	402,138	1.5515
09 Feb 2014 07:35:26	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	233,280	361,746	1.5507
22 Jan 2014 22:23:13	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	207,360	321,829	1.5520
20 Jan 2014 06:44:52	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	181,440	281,673	1.5524
19 Jan 2014 02:48:53	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	155,520	241,796	1.5548
15 Dec 2013 19:45:15	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	129,600	201,933	1.5581
08 Nov 2013 22:33:49	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	103,680	161,266	1.5554
03 Nov 2013 14:14:55	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	77,760	121,011	1.5562
03 Nov 2013 14:14:55	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	51,840	80,521	1.5533
03 Nov 2013 14:14:55	1022874	15992270	hadcm3n_80cf_1980_40_008458386_0	25,920	39,788	1.5350