Task 13347601

Name	hadcm3n_t1jf_1940_40_007442912_0
Workunit	7640415
Created	8 Sep 2011, 22:31:44 UTC
Sent	8 Sep 2011, 22:36:41 UTC
Report deadline	9 Dec 2011, 6:03:52 UTC
Received	8 Dec 2011, 16:39:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1037595
Run time	80 days 9 hours 35 min 7 sec
CPU time	22 days 1 hours 13 min 49 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	2.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:50:13 (511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:18 (554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:59:42 (12182): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:39:13 (555): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:27:51 (535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1038a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1054000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x840004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x85b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1038a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x812800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1027,0xa0be7540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 07:22:15 (561): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:19:13 (575): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:15 (575): No heartbeat from core client for 30 sec - exiting 18:26:59 (2050): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:34 (2145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:17 (2259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:49 (2309): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:59:41 (2328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:39:43 (546): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:33:50 (15760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:11:14 (545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:06:16 (525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(918,0xa0436540) malloc: * error for object 0x822c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=922, selfPID=922, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(651,0xa0436540) malloc: * error for object 0x1019e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 10:41:42 (42595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:41:44 (42595): No heartbeat from core client for 30 sec - exiting 10:41:45 (42595): No heartbeat from core client for 30 sec - exiting 10:41:46 (42595): No heartbeat from core client for 30 sec - exiting 10:41:47 (42595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 16:43:16 (581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:18 (581): No heartbeat from core client for 30 sec - exiting 16:43:19 (581): No heartbeat from core client for 30 sec - exiting 16:43:20 (581): No heartbeat from core client for 30 sec - exiting 16:43:21 (581): No heartbeat from core client for 30 sec - exiting 16:43:22 (581): No heartbeat from core client for 30 sec - exiting 16:43:23 (581): No heartbeat from core client for 30 sec - exiting 16:43:24 (581): No heartbeat from core client for 30 sec - exiting 16:43:25 (581): No heartbeat from core client for 30 sec - exiting 16:43:26 (581): No heartbeat from core client for 30 sec - exiting 16:43:27 (581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:23:53 (612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:00:21 (36270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:20:45 (613): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:08:31 (608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:07:31 (15288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:46:09 (623): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:10 (623): No heartbeat from core client for 30 sec - exiting 18:46:11 (623): No heartbeat from core client for 30 sec - exiting 18:46:12 (623): No heartbeat from core client for 30 sec - exiting execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135925) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5625, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Dec 2011 23:57:45	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	829,440	1,881,899	2.2689
03 Dec 2011 12:37:08	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	803,520	1,822,277	2.2679
29 Nov 2011 09:11:48	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	777,600	1,763,187	2.2675
25 Nov 2011 22:26:42	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	751,680	1,703,222	2.2659
23 Nov 2011 05:46:57	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	725,760	1,641,351	2.2616
20 Nov 2011 20:19:56	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	699,840	1,579,510	2.2570
17 Nov 2011 11:19:02	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	673,920	1,518,067	2.2526
15 Nov 2011 18:09:33	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	648,000	1,456,524	2.2477
15 Nov 2011 18:09:33	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	622,080	1,394,258	2.2413
08 Nov 2011 13:03:32	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	596,160	1,332,458	2.2351
06 Nov 2011 04:53:57	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	570,240	1,270,325	2.2277
04 Nov 2011 14:03:37	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	544,320	1,208,667	2.2205
02 Nov 2011 09:54:59	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	518,400	1,145,862	2.2104
31 Oct 2011 19:38:20	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	492,480	1,086,392	2.2060
31 Oct 2011 18:29:33	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	466,560	1,027,727	2.2028
31 Oct 2011 13:06:43	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	440,640	970,364	2.2022
31 Oct 2011 13:06:43	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	414,720	912,885	2.2012
18 Oct 2011 12:01:11	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	388,800	856,848	2.2038
15 Oct 2011 16:19:15	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	362,880	800,640	2.2063
13 Oct 2011 00:20:55	1037595	13347601	hadcm3n_t1jf_1940_40_007442912_0	336,960	743,659	2.2070