Task 15719745

Name	hadcm3n_4fb3_1940_40_008302640_3
Workunit	8453775
Created	10 Apr 2013, 4:52:30 UTC
Sent	10 Apr 2013, 4:52:57 UTC
Report deadline	10 Jul 2013, 12:20:08 UTC
Received	21 Apr 2013, 9:43:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1185419
Run time	8 days 4 hours 46 min 52 sec
CPU time	5 days 23 hours 32 min 32 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(1440,0xaced22c0) malloc: * error for object 0x862604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1440,0xaced22c0) malloc: * error for object 0x862600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1440,0xaced22c0) malloc: * error for object 0x862604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1440,0xaced22c0) malloc: * error for object 0x862600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1440,0xaced22c0) malloc: * error for object 0x862604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(865,0xaced22c0) malloc: * error for object 0x1350000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2013 21:53:37	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	622,080	581,167	0.9342
20 Apr 2013 13:35:06	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	596,160	556,864	0.9341
20 Apr 2013 04:37:35	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	570,240	532,667	0.9341
19 Apr 2013 18:02:08	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	544,320	508,389	0.9340
19 Apr 2013 07:13:07	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	518,400	484,191	0.9340
18 Apr 2013 19:41:54	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	492,480	460,272	0.9346
18 Apr 2013 08:26:09	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	466,560	435,865	0.9342
17 Apr 2013 19:36:36	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	440,640	410,905	0.9325
17 Apr 2013 08:56:51	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	414,720	386,610	0.9322
16 Apr 2013 23:13:28	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	388,800	362,201	0.9316
16 Apr 2013 08:09:30	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	362,880	338,270	0.9322
15 Apr 2013 22:42:31	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	336,960	314,034	0.9320
15 Apr 2013 14:17:17	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	311,040	289,999	0.9324
15 Apr 2013 01:17:59	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	285,120	265,833	0.9324
14 Apr 2013 13:50:16	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	259,200	241,267	0.9308
14 Apr 2013 04:55:19	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	233,280	217,311	0.9315
13 Apr 2013 15:50:55	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	207,360	193,332	0.9323
13 Apr 2013 04:57:17	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	181,440	168,891	0.9308
12 Apr 2013 19:57:27	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	155,520	144,306	0.9279
12 Apr 2013 07:54:01	1185419	15719745	hadcm3n_4fb3_1940_40_008302640_3	129,600	120,426	0.9292