Task 14364984

Name	hadcm3n_o6dr_2020_40_007857632_4
Workunit	8012744
Created	5 Apr 2012, 15:12:09 UTC
Sent	5 Apr 2012, 15:35:02 UTC
Report deadline	5 Jul 2012, 23:02:13 UTC
Received	11 May 2012, 11:06:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	4 (0x00000004) Unknown error code
Computer ID	1072552
Run time	8 days 1 hours 10 min 57 sec
CPU time	6 days 17 hours 47 min 31 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process got signal 4 </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:10:27 (96065): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:10:31 (96065): No heartbeat from core client for 30 sec - exiting 11:10:32 (96065): No heartbeat from core client for 30 sec - exiting 11:10:33 (96065): No heartbeat from core client for 30 sec - exiting 11:10:34 (96065): No heartbeat from core client for 30 sec - exiting 11:10:35 (96065): No heartbeat from core client for 30 sec - exiting 11:10:36 (96065): No heartbeat from core client for 30 sec - exiting 11:10:37 (96065): No heartbeat from core client for 30 sec - exiting 11:10:38 (96065): No heartbeat from core client for 30 sec - exiting 11:10:39 (96065): No heartbeat from core client for 30 sec - exiting 11:10:40 (96065): No heartbeat from core client for 30 sec - exiting 11:10:41 (96065): No heartbeat from core client for 30 sec - exiting 11:10:42 (96065): No heartbeat from core client for 30 sec - exiting 11:10:43 (96065): No heartbeat from core client for 30 sec - exiting 11:10:44 (96065): No heartbeat from core client for 30 sec - exiting 11:10:45 (96065): No heartbeat from core client for 30 sec - exiting 11:10:46 (96065): No heartbeat from core client for 30 sec - exiting 11:10:47 (96065): No heartbeat from core client for 30 sec - exiting 11:10:48 (96065): No heartbeat from core client for 30 sec - exiting 11:10:49 (96065): No heartbeat from core client for 30 sec - exiting 11:10:51 (96065): No heartbeat from core client for 30 sec - exiting 11:10:52 (96065): No heartbeat from core client for 30 sec - exiting 11:10:53 (96065): No heartbeat from core client for 30 sec - exiting 11:10:54 (96065): No heartbeat from core client for 30 sec - exiting 11:10:55 (96065): No heartbeat from core client for 30 sec - exiting 11:10:56 (96065): No heartbeat from core client for 30 sec - exiting 11:10:57 (96065): No heartbeat from core client for 30 sec - exiting 11:10:58 (96065): No heartbeat from core client for 30 sec - exiting 11:10:59 (96065): No heartbeat from core client for 30 sec - exiting 11:11:00 (96065): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:51:50 (20941): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:52 (20941): No heartbeat from core client for 30 sec - exiting 13:51:53 (20941): No heartbeat from core client for 30 sec - exiting 13:51:54 (20941): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:22:43 (766): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:40:20 (8196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:21 (8196): No heartbeat from core client for 30 sec - exiting 10:03:32 (15539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:34 (15539): No heartbeat from core client for 30 sec - exiting 10:03:36 (15539): No heartbeat from core client for 30 sec - exiting 10:03:37 (15539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:57:44 (87618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:57:45 (87618): No heartbeat from core client for 30 sec - exiting 16:57:46 (87618): No heartbeat from core client for 30 sec - exiting 16:57:47 (87618): No heartbeat from core client for 30 sec - exiting 17:08:01 (39666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:08:02 (39666): No heartbeat from core client for 30 sec - exiting 17:08:03 (39666): No heartbeat from core client for 30 sec - exiting 17:08:04 (39666): No heartbeat from core client for 30 sec - exiting 17:08:05 (39666): No heartbeat from core client for 30 sec - exiting 17:08:06 (39666): No heartbeat from core client for 30 sec - exiting 17:08:08 (39666): No heartbeat from core client for 30 sec - exiting 17:08:09 (39666): No heartbeat from core client for 30 sec - exiting 17:08:10 (39666): No heartbeat from core client for 30 sec - exiting 17:08:11 (39666): No heartbeat from core client for 30 sec - exiting 17:08:12 (39666): No heartbeat from core client for 30 sec - exiting 17:08:13 (39666): No heartbeat from core client for 30 sec - exiting 17:08:14 (39666): No heartbeat from core client for 30 sec - exiting 17:08:15 (39666): No heartbeat from core client for 30 sec - exiting 17:08:16 (39666): No heartbeat from core client for 30 sec - exiting 17:08:17 (39666): No heartbeat from core client for 30 sec - exiting 17:08:18 (39666): No heartbeat from core client for 30 sec - exiting 17:08:19 (39666): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 17:39:36 (39859): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:37 (39859): No heartbeat from core client for 30 sec - exiting 17:39:38 (39859): No heartbeat from core client for 30 sec - exiting 17:39:39 (39859): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:56:51 (97698): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:56:52 (97698): No heartbeat from core client for 30 sec - exiting 00:56:54 (97698): No heartbeat from core client for 30 sec - exiting 00:56:55 (97698): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97860,0xac85c2c0) malloc: * error for object 0xb46e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 02:53:02 (97860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:03 (97860): No heartbeat from core client for 30 sec - exiting 02:53:04 (97860): No heartbeat from core client for 30 sec - exiting 02:53:05 (97860): No heartbeat from core client for 30 sec - exiting 02:53:06 (97860): No heartbeat from core client for 30 sec - exiting 03:09:12 (99919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:13 (99919): No heartbeat from core client for 30 sec - exiting 03:09:14 (99919): No heartbeat from core client for 30 sec - exiting 03:09:15 (99919): No heartbeat from core client for 30 sec - exiting 03:09:16 (99919): No heartbeat from core client for 30 sec - exiting 03:09:17 (99919): No heartbeat from core client for 30 sec - exiting 03:09:18 (99919): No heartbeat from core client for 30 sec - exiting 03:09:19 (99919): No heartbeat from core client for 30 sec - exiting 03:09:20 (99919): No heartbeat from core client for 30 sec - exiting 03:09:21 (99919): No heartbeat from core client for 30 sec - exiting 03:09:23 (99919): No heartbeat from core client for 30 sec - exiting 03:09:24 (99919): No heartbeat from core client for 30 sec - exiting 03:09:25 (99919): No heartbeat from core client for 30 sec - exiting 03:09:26 (99919): No heartbeat from core client for 30 sec - exiting 03:09:27 (99919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 May 2012 23:50:12	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	518,400	575,040	1.1093
10 May 2012 00:41:04	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	492,480	546,192	1.1091
05 May 2012 11:57:28	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	466,560	517,854	1.1099
05 May 2012 00:12:26	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	440,640	489,073	1.1099
04 May 2012 15:43:16	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	414,720	460,197	1.1097
03 May 2012 22:48:55	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	388,800	431,323	1.1094
03 May 2012 07:38:57	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	362,880	402,667	1.1096
29 Apr 2012 02:48:53	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	336,960	373,884	1.1096
28 Apr 2012 15:35:33	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	311,040	345,164	1.1097
27 Apr 2012 10:44:10	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	285,120	316,555	1.1103
25 Apr 2012 18:11:42	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	259,200	287,808	1.1104
22 Apr 2012 07:48:48	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	233,280	259,063	1.1105
21 Apr 2012 14:48:08	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	207,360	230,204	1.1102
21 Apr 2012 05:37:37	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	181,440	201,168	1.1087
20 Apr 2012 16:44:39	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	155,520	172,453	1.1089
19 Apr 2012 22:09:25	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	129,600	143,513	1.1074
19 Apr 2012 10:26:51	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	103,680	114,991	1.1091
07 Apr 2012 23:33:48	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	77,760	86,110	1.1074
07 Apr 2012 05:13:12	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	51,840	57,484	1.1089
06 Apr 2012 10:38:01	1072552	14364984	hadcm3n_o6dr_2020_40_007857632_4	25,920	28,722	1.1081