Task 13026614

Name	hadcm3n_t63j_1940_40_007316466_1
Workunit	7513896
Created	29 Jun 2011, 3:26:02 UTC
Sent	29 Jun 2011, 3:32:58 UTC
Report deadline	28 Sep 2011, 11:00:09 UTC
Received	9 Dec 2011, 22:53:27 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1022874
Run time	22 days 21 hours 48 min 3 sec
CPU time	17 days 18 hours 34 min 50 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	3.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... 08:48:33 (4928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:35 (4928): No heartbeat from core client for 30 sec - exiting 08:48:36 (4928): No heartbeat from core client for 30 sec - exiting 08:48:37 (4928): No heartbeat from core client for 30 sec - exiting 08:48:38 (4928): No heartbeat from core client for 30 sec - exiting 08:48:39 (4928): No heartbeat from core client for 30 sec - exiting 08:48:40 (4928): No heartbeat from core client for 30 sec - exiting 08:48:41 (4928): No heartbeat from core client for 30 sec - exiting 08:48:42 (4928): No heartbeat from core client for 30 sec - exiting 08:48:43 (4928): No heartbeat from core client for 30 sec - exiting 08:48:44 (4928): No heartbeat from core client for 30 sec - exiting 18:17:31 (24066): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x104fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x829004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x829004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x829004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(288,0xa054c540) malloc: * error for object 0x844600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=290, selfPID=290, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x859004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x182ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x859004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3641,0xa054c540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(15008,0xa054c540) malloc: * error for object 0x1030604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15008,0xa054c540) malloc: * error for object 0x847604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15008,0xa054c540) malloc: * error for object 0x1030604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15008,0xa054c540) malloc: * error for object 0x1015e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(292,0xa054c540) malloc: * error for object 0x843604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(292,0xa054c540) malloc: * error for object 0x843604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(292,0xa054c540) malloc: * error for object 0x1034604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1040604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x80a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x80a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x80a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x80a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1010e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa054c540) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:53:32 (21519): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:34 (21519): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:14:00 (17676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:14:01 (17676): No heartbeat from core client for 30 sec - exiting 17:14:02 (17676): No heartbeat from core client for 30 sec - exiting 21:34:06 (23799): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish 10:02:33 (33284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:34 (33284): No heartbeat from core client for 30 sec - exiting 10:02:35 (33284): No heartbeat from core client for 30 sec - exiting 10:02:36 (33284): No heartbeat from core client for 30 sec - exiting 10:02:37 (33284): No heartbeat from core client for 30 sec - exiting 10:02:38 (33284): No heartbeat from core client for 30 sec - exiting 10:02:39 (33284): No heartbeat from core client for 30 sec - exiting 10:02:40 (33284): No heartbeat from core client for 30 sec - exiting 10:02:41 (33284): No heartbeat from core client for 30 sec - exiting 10:02:42 (33284): No heartbeat from core client for 30 sec - exiting 10:02:43 (33284): No heartbeat from core client for 30 sec - exiting 10:02:44 (33284): No heartbeat from core client for 30 sec - exiting 10:02:45 (33284): No heartbeat from core client for 30 sec - exiting 10:02:46 (33284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:23:35 (271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Dec 2011 21:58:17	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	1,036,800	1,535,154	1.4807
09 Dec 2011 21:58:17	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	1,010,880	1,500,848	1.4847
09 Dec 2011 21:58:17	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	984,960	1,466,284	1.4887
09 Dec 2011 21:58:17	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	959,040	1,429,554	1.4906
04 Dec 2011 06:09:15	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	933,120	1,389,175	1.4887
22 Nov 2011 20:59:14	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	907,200	1,348,337	1.4863
22 Nov 2011 06:58:18	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	881,280	1,308,030	1.4842
19 Nov 2011 23:40:43	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	855,360	1,267,393	1.4817
07 Nov 2011 16:03:48	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	829,440	1,226,650	1.4789
05 Nov 2011 14:52:31	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	803,520	1,186,328	1.4764
08 Oct 2011 17:47:05	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	777,600	1,145,728	1.4734
05 Oct 2011 21:57:58	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	751,680	1,105,496	1.4707
30 Sep 2011 04:16:59	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	725,760	1,065,199	1.4677
29 Sep 2011 20:39:37	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	699,840	1,025,067	1.4647
27 Sep 2011 22:42:00	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	673,920	984,790	1.4613
27 Sep 2011 01:02:53	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	648,000	944,457	1.4575
24 Sep 2011 08:56:33	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	622,080	904,665	1.4543
21 Sep 2011 02:37:41	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	596,160	867,909	1.4558
19 Sep 2011 17:46:28	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	570,240	831,010	1.4573
18 Sep 2011 11:34:44	1022874	13026614	hadcm3n_t63j_1940_40_007316466_1	544,320	836,989	1.5377