Task 19133578

Name	hadcm3n_sdof_198012_480_010222164_0
Workunit	10222164
Created	4 Dec 2015, 19:30:16 UTC
Sent	5 Dec 2015, 0:16:05 UTC
Report deadline	16 Nov 2016, 5:36:05 UTC
Received	8 Dec 2015, 19:19:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1340635
Run time	2 days 13 hours 36 min 43 sec
CPU time	2 days 11 hours 21 min 16 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	4.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.12</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x850204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x850200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x281ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x281ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x182fe04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x302aa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x3046004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x3046000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x834c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x182fe04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x283be04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x283be00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x302aa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14892,0xa436b000) malloc: * error for object 0x81e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x302b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x2828604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x1830804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x302b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x481f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15285,0xa436b000) malloc: * error for object 0x3011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15344,0xa436b000) malloc: * error for object 0x1016200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15562,0xa436b000) malloc: * error for object 0x1029e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel x86-64h Haswell (32-bit executable) System version: Macintosh OS 10.11.1 build 15B42 Wed Dec 9 04:47:18 2015 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd88 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/7/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9004a000 - 0x9004cfff /usr/lib/system/libsystem_secinit.dylib 0x90174000 - 0x90174fff /usr/lib/system/liblaunch.dylib 0x9029e000 - 0x902c7fff /usr/lib/system/libsystem_info.dylib 0x905a1000 - 0x905acfff /usr/lib/system/libcommonCrypto.dylib 0x905cc000 - 0x905cefff /usr/lib/system/libquarantine.dylib 0x905cf000 - 0x905d9fff /usr/lib/system/libsystem_notify.dylib 0x90745000 - 0x9076ffff /usr/lib/system/libdispatch.dylib 0x91027000 - 0x91038fff /usr/lib/system/libsystem_trace.dylib 0x912e1000 - 0x912effff /usr/lib/libz.1.dylib 0x91493000 - 0x9149cfff /usr/lib/system/libcopyfile.dylib 0x91a3c000 - 0x91a40fff /usr/lib/system/libcache.dylib 0x91a5c000 - 0x91a61fff /usr/lib/system/libcompiler_rt.dylib 0x920c1000 - 0x920c4fff /usr/lib/system/libdyld.dylib 0x92916000 - 0x9292bfff /usr/lib/system/libsystem_coretls.dylib 0x9301d000 - 0x9301dfff /usr/lib/system/libunc.dylib 0x930f3000 - 0x93113fff /usr/lib/system/libsystem_kernel.dylib 0x94f03000 - 0x94f6afff /usr/lib/system/libcorecrypto.dylib 0x96261000 - 0x9627afff /usr/lib/system/libsystem_asl.dylib 0x96802000 - 0x9680afff /usr/lib/system/libsystem_dnssd.dylib 0x96b88000 - 0x96c1dfff /usr/lib/system/libsystem_c.dylib 0x96c59000 - 0x96c59fff /usr/lib/system/libkeymgr.dylib 0x971f0000 - 0x971f9fff /usr/lib/system/libsystem_networkextension.dylib 0x97c4e000 - 0x97cb4fff /usr/lib/system/libsystem_network.dylib 0x97cc1000 - 0x97ce4fff /usr/lib/libc++abi.dylib 0x97ce8000 - 0x97d3bfff /usr/lib/libstdc++.6.dylib 0x98fe2000 - 0x98fe7fff /usr/lib/system/libmacho.dylib 0x9996b000 - 0x9996cfff /usr/lib/system/libremovefile.dylib 0x9b195000 - 0x9b1bdfff /usr/lib/system/libxpc.dylib 0x9b876000 - 0x9b879fff /usr/lib/system/libsystem_sandbox.dylib 0x9b881000 - 0x9b89afff /usr/lib/system/libsystem_malloc.dylib 0x9b89b000 - 0x9b8a1fff /usr/lib/system/libsystem_platform.dylib 0x9b8a2000 - 0x9b8a4fff /usr/lib/system/libsystem_configuration.dylib 0x9c01a000 - 0x9c020fff /usr/lib/system/libunwind.dylib 0x9c021000 - 0x9c054fff /usr/lib/system/libsystem_m.dylib 0x9c2ac000 - 0x9c2adfff /usr/lib/libSystem.B.dylib 0x9e4a4000 - 0x9e4a6fff /usr/lib/system/libsystem_coreservices.dylib 0x9e5cc000 - 0x9e5d4fff /usr/lib/system/libsystem_pthread.dylib 0x9ecfb000 - 0x9ecfbfff /usr/lib/system/libsystem_blocks.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Dec 2015 19:23:12	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	259,200	213,243	0.8227
08 Dec 2015 09:45:18	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	233,280	192,022	0.8231
08 Dec 2015 02:36:20	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	207,360	170,147	0.8205
07 Dec 2015 18:18:03	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	181,440	147,730	0.8142
07 Dec 2015 10:35:20	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	155,520	126,012	0.8103
07 Dec 2015 01:05:51	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	129,600	104,420	0.8057
06 Dec 2015 12:16:11	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	103,680	81,997	0.7909
06 Dec 2015 02:33:14	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	77,760	61,701	0.7935
05 Dec 2015 18:29:41	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	51,840	41,939	0.8090
05 Dec 2015 08:28:50	1340635	19133578	hadcm3n_sdof_198012_480_010222164_0	25,920	20,818	0.8032