Task 17502803

Name	hadcm3n_x0qp_1940_40_009148228_1
Workunit	9278564
Created	27 Nov 2014, 16:02:04 UTC
Sent	27 Nov 2014, 16:06:15 UTC
Report deadline	26 Feb 2015, 23:33:26 UTC
Received	2 Dec 2014, 10:05:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1451375
Run time	4 days 10 hours 13 min 58 sec
CPU time	4 days 1 hours 20 min 34 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.74 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.4.26</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 12:53:27 (98680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:54:44 (2911): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:11 (6312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:55 (10088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:10 (17291): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:10 (20729): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:11 (26792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:02 (31013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:13 (34804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:36:13 (38877): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:13 (42692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:36 (46804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:05:29 (50818): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:04:18 (62352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:33:30 (265): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:07:28 (263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 20:23:02 (4210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:01 (8219): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:10 (11667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:09:08 (19613): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:16:19 (30966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:31:13 (42140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:28 (45919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:49:59 (60871): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:47 (64361): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:49 (68304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:27 (72208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:51 (75834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:46 (79999): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:23:54 (83681): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:47:21 (87203): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:52 (90817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:21 (94910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:57 (99769): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:28 (3717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:49 (18497): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:26:48 (24411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:57 (27824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:13 (31872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:27:12 (36996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:57 (41878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(45837,0xa0bde1d4) malloc: * error for object 0x4829e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(45837,0xa0bde1d4) malloc: * error for object 0x4829e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(45837,0xa0bde1d4) malloc: * error for object 0x2802400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation 15:55:16 (45837): No heartbeat from core client for 30 sec - exiting 15:55:17 (45837): No heartbeat from core client for 30 sec - exiting 15:55:18 (45837): No heartbeat from core client for 30 sec - exiting 16:35:51 (45837): No heartbeat from core client for 30 sec - exiting 16:35:52 (45837): No heartbeat from core client for 30 sec - exiting 16:35:53 (45837): No heartbeat from core client for 30 sec - exiting 16:35:54 (45837): No heartbeat from core client for 30 sec - exiting 16:35:55 (45837): No heartbeat from core client for 30 sec - exiting 18:31:23 (45837): No heartbeat from core client for 30 sec - exiting 18:31:24 (45837): No heartbeat from core client for 30 sec - exiting 18:31:25 (45837): No heartbeat from core client for 30 sec - exiting 18:31:26 (45837): No heartbeat from core client for 30 sec - exiting 18:31:27 (45837): No heartbeat from core client for 30 sec - exiting 18:31:28 (45837): No heartbeat from core client for 30 sec - exiting 21:41:46 (45837): No heartbeat from core client for 30 sec - exiting 21:41:47 (45837): No heartbeat from core client for 30 sec - exiting 21:41:48 (45837): No heartbeat from core client for 30 sec - exiting 21:41:49 (45837): No heartbeat from core client for 30 sec - exiting 21:41:50 (45837): No heartbeat from core client for 30 sec - exiting 21:43:26 (45837): No heartbeat from core client for 30 sec - exiting 21:43:27 (45837): No heartbeat from core client for 30 sec - exiting 21:43:28 (45837): No heartbeat from core client for 30 sec - exiting 21:43:29 (45837): No heartbeat from core client for 30 sec - exiting 21:43:30 (45837): No heartbeat from core client for 30 sec - exiting 21:43:31 (45837): No heartbeat from core client for 30 sec - exiting 21:43:32 (45837): No heartbeat from core client for 30 sec - exiting 00:17:27 (45837): No heartbeat from core client for 30 sec - exiting 00:17:28 (45837): No heartbeat from core client for 30 sec - exiting 00:17:29 (45837): No heartbeat from core client for 30 sec - exiting 00:17:30 (45837): No heartbeat from core client for 30 sec - exiting 00:17:31 (45837): No heartbeat from core client for 30 sec - exiting 00:17:32 (45837): No heartbeat from core client for 30 sec - exiting 00:17:33 (45837): No heartbeat from core client for 30 sec - exiting 00:17:34 (45837): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(253,0xa0bde1d4) malloc: * error for object 0x300de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(253,0xa0bde1d4) malloc: * error for object 0x300f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.10.1 build 14B25 Tue Dec 2 02:04:25 2014 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fe08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/7/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90008000 - 0x9000cfff /usr/lib/system/libcache.dylib 0x9000d000 - 0x9000dfff /usr/lib/system/liblaunch.dylib 0x905f5000 - 0x905fafff /usr/lib/system/libmacho.dylib 0x90669000 - 0x90692fff /usr/lib/system/libsystem_info.dylib 0x91174000 - 0x91175fff /usr/lib/system/libremovefile.dylib 0x9128d000 - 0x91299fff /usr/lib/system/libcommonCrypto.dylib 0x912de000 - 0x91304fff /usr/lib/libc++abi.dylib 0x91305000 - 0x9131cfff /usr/lib/system/libsystem_asl.dylib 0x91bb8000 - 0x91bbbfff /usr/lib/system/libdyld.dylib 0x91bbc000 - 0x91bdbfff /usr/lib/system/libsystem_kernel.dylib 0x926dc000 - 0x92702fff /usr/lib/system/libxpc.dylib 0x93198000 - 0x9319afff /usr/lib/system/libsystem_sandbox.dylib 0x9323d000 - 0x9324efff /usr/lib/system/libsystem_coretls.dylib 0x9324f000 - 0x93282fff /usr/lib/system/libsystem_m.dylib 0x94493000 - 0x9449cfff /usr/lib/system/libcopyfile.dylib 0x94dac000 - 0x94dacfff /usr/lib/system/libunc.dylib 0x94f9d000 - 0x94fa3fff /usr/lib/system/libsystem_platform.dylib 0x9504a000 - 0x9504cfff /usr/lib/system/libsystem_coreservices.dylib 0x9504d000 - 0x9504efff /usr/lib/libSystem.B.dylib 0x95050000 - 0x95052fff /usr/lib/system/libquarantine.dylib 0x953e5000 - 0x9540cfff /usr/lib/system/libdispatch.dylib 0x958e4000 - 0x958e6fff /usr/lib/system/libsystem_secinit.dylib 0x961f7000 - 0x96260fff /usr/lib/system/libcorecrypto.dylib 0x9635e000 - 0x96364fff /usr/lib/system/libsystem_trace.dylib 0x9681c000 - 0x96821fff /usr/lib/system/libcompiler_rt.dylib 0x969ed000 - 0x969edfff /usr/lib/system/libkeymgr.dylib 0x975a2000 - 0x975aafff /usr/lib/system/libsystem_dnssd.dylib 0x976a0000 - 0x97735fff /usr/lib/system/libsystem_c.dylib 0x97737000 - 0x97740fff /usr/lib/system/libsystem_notify.dylib 0x980c9000 - 0x98106fff /usr/lib/system/libsystem_network.dylib 0x982af000 - 0x982b1fff /usr/lib/system/libsystem_configuration.dylib 0x98fcf000 - 0x98fd5fff /usr/lib/system/libsystem_networkextension.dylib 0x996a4000 - 0x996b2fff /usr/lib/libz.1.dylib 0x996c1000 - 0x996c9fff /usr/lib/system/libsystem_pthread.dylib 0x996ca000 - 0x996cbfff /usr/lib/system/libsystem_blocks.dylib 0x999d3000 - 0x99a26fff /usr/lib/libstdc++.6.dylib 0x99ea1000 - 0x99ea8fff /usr/lib/system/libunwind.dylib 0x9a95f000 - 0x9a978fff /usr/lib/system/libsystem_malloc.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Dec 2014 22:37:53	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	259,200	319,473	1.2325
01 Dec 2014 13:15:01	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	233,280	289,151	1.2395
01 Dec 2014 03:56:21	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	207,360	258,233	1.2453
30 Nov 2014 18:22:11	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	181,440	226,551	1.2486
30 Nov 2014 08:23:41	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	155,520	193,154	1.2420
29 Nov 2014 22:43:28	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	129,600	160,688	1.2399
29 Nov 2014 13:42:40	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	103,680	131,125	1.2647
29 Nov 2014 02:29:16	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	77,760	100,847	1.2969
28 Nov 2014 16:17:32	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	51,840	69,332	1.3374
28 Nov 2014 05:56:35	1313549	17502803	hadcm3n_x0qp_1940_40_009148228_1	25,920	35,114	1.3547