Task 14850485

Name	hadcm3n_t2sm_1940_40_007956011_3
Workunit	8111123
Created	30 Jun 2012, 15:29:41 UTC
Sent	30 Jun 2012, 15:30:04 UTC
Report deadline	29 Sep 2012, 22:57:15 UTC
Received	3 Jul 2012, 22:27:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1202614
Run time	2 days 19 hours 51 min 20 sec
CPU time	2 days 4 hours 29 min 58 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	3.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:33:36 (17802): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:37:52 (32664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:54 (32664): No heartbeat from core client for 30 sec - exiting 22:37:55 (32664): No heartbeat from core client for 30 sec - exiting 22:37:56 (32664): No heartbeat from core client for 30 sec - exiting 22:37:57 (32664): No heartbeat from core client for 30 sec - exiting 22:37:58 (32664): No heartbeat from core client for 30 sec - exiting 22:37:59 (32664): No heartbeat from core client for 30 sec - exiting 22:38:00 (32664): No heartbeat from core client for 30 sec - exiting 22:38:01 (32664): No heartbeat from core client for 30 sec - exiting 22:38:02 (32664): No heartbeat from core client for 30 sec - exiting 22:38:03 (32664): No heartbeat from core client for 30 sec - exiting 22:38:04 (32664): No heartbeat from core client for 30 sec - exiting 22:38:05 (32664): No heartbeat from core client for 30 sec - exiting 22:38:06 (32664): No heartbeat from core client for 30 sec - exiting 22:38:07 (32664): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 22:45:58 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=33626, iMonCtr=1 Model crash detected, will try to restart... 22:47:32 (33626): No heartbeat from core client for 30 sec - exiting 22:47:38 (33626): No heartbeat from core client for 30 sec - exiting 22:47:39 (33626): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 22:51:43 2012 Thread 0 Crashed: Thread 1: Suspended CPDN Monitor - Suspend request from BOINC... Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=33752, iMonCtr=1 Model crash detected, will try to restart... 22:56:22 (33752): No heartbeat from core client for 30 sec - exiting 22:56:26 (33752): No heartbeat from core client for 30 sec - exiting 22:56:27 (33752): No heartbeat from core client for 30 sec - exiting 22:56:28 (33752): No heartbeat from core client for 30 sec - exiting 22:56:29 (33752): No heartbeat from core client for 30 sec - exiting 22:56:30 (33752): No heartbeat from core client for 30 sec - exiting 22:56:31 (33752): No heartbeat from core client for 30 sec - exiting 22:56:32 (33752): No heartbeat from core client for 30 sec - exiting 22:56:33 (33752): No heartbeat from core client for 30 sec - exiting 22:56:34 (33752): No heartbeat from core client for 30 sec - exiting 22:56:35 (33752): No heartbeat from core client for 30 sec - exiting 22:56:36 (33752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 23:00:30 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=33857, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 23:07:50 2012 Thread 0 Crashed: Thread 1: 23:08:31 (33857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 23:12:40 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=33976, iMonCtr=1 Model crash detected, will try to restart... 23:16:43 (33976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:45 (33976): No heartbeat from core client for 30 sec - exiting 23:16:46 (33976): No heartbeat from core client for 30 sec - exiting 23:16:47 (33976): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 23:19:57 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34072, iMonCtr=1 Model crash detected, will try to restart... 23:22:34 (34072): No heartbeat from core client for 30 sec - exiting 23:22:37 (34072): No heartbeat from core client for 30 sec - exiting 23:22:38 (34072): No heartbeat from core client for 30 sec - exiting 23:22:39 (34072): No heartbeat from core client for 30 sec - exiting 23:22:40 (34072): No heartbeat from core client for 30 sec - exiting 23:22:41 (34072): No heartbeat from core client for 30 sec - exiting 23:22:42 (34072): No heartbeat from core client for 30 sec - exiting 23:22:43 (34072): No heartbeat from core client for 30 sec - exiting 23:22:44 (34072): No heartbeat from core client for 30 sec - exiting 23:22:45 (34072): No heartbeat from core client for 30 sec - exiting 23:22:46 (34072): No heartbeat from core client for 30 sec - exiting 23:22:47 (34072): No heartbeat from core client for 30 sec - exiting 23:22:48 (34072): No heartbeat from core client for 30 sec - exiting 23:22:49 (34072): No heartbeat from core client for 30 sec - exiting 23:22:50 (34072): No heartbeat from core client for 30 sec - exiting 23:22:51 (34072): No heartbeat from core client for 30 sec - exiting 23:22:52 (34072): No heartbeat from core client for 30 sec - exiting 23:22:53 (34072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Tue Jul 3 23:25:38 2012 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffcd08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x9074b000 - 0x9077afff /usr/lib/system/libsystem_info.dylib 0x91069000 - 0x9106afff /usr/lib/system/libremovefile.dylib 0x91231000 - 0x91238fff /usr/lib/system/libsystem_dnssd.dylib 0x91eba000 - 0x91ebbfff /usr/lib/system/libsystem_sandbox.dylib 0x92be9000 - 0x92bf1fff /usr/lib/system/liblaunch.dylib 0x938ca000 - 0x938cbfff /usr/lib/system/libsystem_blocks.dylib 0x93ca2000 - 0x93ca2fff /usr/lib/system/libkeymgr.dylib 0x93d2b000 - 0x93d6efff /usr/lib/system/libcommonCrypto.dylib 0x940ac000 - 0x940b4fff /usr/lib/system/libunwind.dylib 0x940b5000 - 0x940d3fff /usr/lib/system/libsystem_kernel.dylib 0x94264000 - 0x9427afff /usr/lib/system/libxpc.dylib 0x94587000 - 0x94588fff /usr/lib/system/libquarantine.dylib 0x94589000 - 0x9458cfff /usr/lib/system/libmathCommon.A.dylib 0x94c24000 - 0x94c28fff /usr/lib/system/libcache.dylib 0x95883000 - 0x95883fff /usr/lib/system/libdnsinfo.dylib 0x95e21000 - 0x95e23fff /usr/lib/system/libdyld.dylib 0x961f3000 - 0x96201fff /usr/lib/system/libdispatch.dylib 0x96d35000 - 0x96d3dfff /usr/lib/system/libcopyfile.dylib 0x9720d000 - 0x972d8fff /usr/lib/system/libsystem_c.dylib 0x98eff000 - 0x98f02fff /usr/lib/system/libcompiler_rt.dylib 0x99b77000 - 0x99b7bfff /usr/lib/system/libsystem_network.dylib 0x9a675000 - 0x9a6a3fff /usr/lib/libSystem.B.dylib 0x9b8a5000 - 0x9b8a6fff /usr/lib/system/libunc.dylib 0x9bf95000 - 0x9bfa3fff /usr/lib/libz.1.dylib 0x9bfa4000 - 0x9bfabfff /usr/lib/system/libsystem_notify.dylib 0x9c077000 - 0x9c07cfff /usr/lib/system/libmacho.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34146, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Jul 2012 19:33:16	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	155,520	186,532	1.1994
03 Jul 2012 08:07:58	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	129,600	154,618	1.1930
02 Jul 2012 13:02:32	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	103,680	122,481	1.1813
02 Jul 2012 12:01:04	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	77,760	92,254	1.1864
02 Jul 2012 12:01:04	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	51,840	61,612	1.1885
02 Jul 2012 12:01:04	1202614	14850485	hadcm3n_t2sm_1940_40_007956011_3	25,920	31,027	1.1970