Task 15821220

Name	hadcm3n_o1pf_1980_40_008385818_0
Workunit	8536677
Created	3 Jun 2013, 4:27:02 UTC
Sent	9 Jun 2013, 9:18:16 UTC
Report deadline	8 Sep 2013, 16:45:27 UTC
Received	13 Jun 2013, 23:22:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1242102
Run time	3 days 4 hours 44 min 57 sec
CPU time	2 days 15 hours 53 min 25 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 02:36:56 (2349): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:27 (2645): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:28 (2645): No heartbeat from core client for 30 sec - exiting 02:55:29 (2645): No heartbeat from core client for 30 sec - exiting 02:55:30 (2645): No heartbeat from core client for 30 sec - exiting 02:56:50 (2695): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:51 (2695): No heartbeat from core client for 30 sec - exiting 02:56:52 (2695): No heartbeat from core client for 30 sec - exiting 02:56:53 (2695): No heartbeat from core client for 30 sec - exiting 02:56:54 (2695): No heartbeat from core client for 30 sec - exiting 03:57:17 (2702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:18 (2702): No heartbeat from core client for 30 sec - exiting 03:57:19 (2702): No heartbeat from core client for 30 sec - exiting 03:57:20 (2702): No heartbeat from core client for 30 sec - exiting 03:57:21 (2702): No heartbeat from core client for 30 sec - exiting 04:13:10 (2926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:12 (2926): No heartbeat from core client for 30 sec - exiting 04:14:38 (2965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:37:07 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:39:56 (3047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:27 (3064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:28 (3064): No heartbeat from core client for 30 sec - exiting 04:44:29 (3064): No heartbeat from core client for 30 sec - exiting 04:44:30 (3064): No heartbeat from core client for 30 sec - exiting 04:44:31 (3064): No heartbeat from core client for 30 sec - exiting 04:51:00 (3082): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:01 (3082): No heartbeat from core client for 30 sec - exiting 04:53:50 (3103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:06 (3119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:22 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:23 (3132): No heartbeat from core client for 30 sec - exiting 04:58:24 (3132): No heartbeat from core client for 30 sec - exiting 05:15:36 (3146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:07 (3197): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:26:31 (3226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:26:32 (3226): No heartbeat from core client for 30 sec - exiting 05:26:33 (3226): No heartbeat from core client for 30 sec - exiting 05:31:32 (3248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:42:48 (3260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:37 (3299): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:00 (3305): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:01 (3305): No heartbeat from core client for 30 sec - exiting 05:57:57 (3333): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:57:58 (3333): No heartbeat from core client for 30 sec - exiting 05:57:59 (3333): No heartbeat from core client for 30 sec - exiting 05:58:00 (3333): No heartbeat from core client for 30 sec - exiting 06:09:19 (3348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:21 (3348): No heartbeat from core client for 30 sec - exiting 06:17:16 (3387): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:34 (3409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:36 (3409): No heartbeat from core client for 30 sec - exiting 06:18:37 (3409): No heartbeat from core client for 30 sec - exiting 06:18:38 (3409): No heartbeat from core client for 30 sec - exiting 06:18:39 (3409): No heartbeat from core client for 30 sec - exiting 06:22:46 (3428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:58 (3437): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:01 (3461): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:05 (3491): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:29 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:30 (3568): No heartbeat from core client for 30 sec - exiting 07:10:30 (3572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:32 (3572): No heartbeat from core client for 30 sec - exiting 07:14:27 (3618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:18 (3629): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:01 (3658): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:03 (3658): No heartbeat from core client for 30 sec - exiting 07:20:04 (3658): No heartbeat from core client for 30 sec - exiting 07:20:05 (3658): No heartbeat from core client for 30 sec - exiting 07:20:06 (3658): No heartbeat from core client for 30 sec - exiting 07:33:35 (3666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:36 (3666): No heartbeat from core client for 30 sec - exiting 07:33:37 (3666): No heartbeat from core client for 30 sec - exiting 07:33:38 (3666): No heartbeat from core client for 30 sec - exiting 07:40:33 (3726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:22 (3742): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:24 (3742): No heartbeat from core client for 30 sec - exiting 07:43:25 (3742): No heartbeat from core client for 30 sec - exiting 07:43:26 (3742): No heartbeat from core client for 30 sec - exiting 07:43:27 (3742): No heartbeat from core client for 30 sec - exiting 07:50:24 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:19 (3782): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:21 (3782): No heartbeat from core client for 30 sec - exiting 08:09:22 (3782): No heartbeat from core client for 30 sec - exiting 08:09:23 (3782): No heartbeat from core client for 30 sec - exiting 08:28:48 (3834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:49 (3834): No heartbeat from core client for 30 sec - exiting 08:28:50 (3834): No heartbeat from core client for 30 sec - exiting 08:30:09 (3898): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:31:26 (3913): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:32:39 (3919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:33:59 (3933): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 08:43:31 (3939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:16 (3970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:18 (3970): No heartbeat from core client for 30 sec - exiting 08:59:26 (4005): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:50 (4023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:35 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:37 (4032): No heartbeat from core client for 30 sec - exiting 09:03:38 (4032): No heartbeat from core client for 30 sec - exiting 09:03:39 (4032): No heartbeat from core client for 30 sec - exiting 09:03:40 (4032): No heartbeat from core client for 30 sec - exiting 09:03:41 (4032): No heartbeat from core client for 30 sec - exiting 09:06:46 (4045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:14 (4063): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:15 (4063): No heartbeat from core client for 30 sec - exiting 09:34:16 (4063): No heartbeat from core client for 30 sec - exiting 09:34:17 (4063): No heartbeat from core client for 30 sec - exiting 09:34:18 (4063): No heartbeat from core client for 30 sec - exiting 09:35:07 (4139): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:03 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:08 (4183): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:09 (4183): No heartbeat from core client for 30 sec - exiting 10:03:00 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:00 (4235): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:01 (4235): No heartbeat from core client for 30 sec - exiting 10:22:21 (4295): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:35:01 (4299): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:37:45 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:37:47 (4360): No heartbeat from core client for 30 sec - exiting 10:37:48 (4360): No heartbeat from core client for 30 sec - exiting 10:37:49 (4360): No heartbeat from core client for 30 sec - exiting 10:37:50 (4360): No heartbeat from core client for 30 sec - exiting 10:37:51 (4360): No heartbeat from core client for 30 sec - exiting 10:37:52 (4360): No heartbeat from core client for 30 sec - exiting 10:37:53 (4360): No heartbeat from core client for 30 sec - exiting 10:37:54 (4360): No heartbeat from core client for 30 sec - exiting 11:04:23 (4369): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:23 (4434): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:44 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:45 (4484): No heartbeat from core client for 30 sec - exiting 11:24:46 (4484): No heartbeat from core client for 30 sec - exiting 11:24:47 (4484): No heartbeat from core client for 30 sec - exiting 11:24:48 (4484): No heartbeat from core client for 30 sec - exiting 11:24:49 (4484): No heartbeat from core client for 30 sec - exiting 11:35:13 (4496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:09 (4535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:32 (4582): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:33 (4582): No heartbeat from core client for 30 sec - exiting 11:51:34 (4582): No heartbeat from core client for 30 sec - exiting 11:51:35 (4582): No heartbeat from core client for 30 sec - exiting 11:59:04 (4586): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:28 (4618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:47 (4627): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:48 (4627): No heartbeat from core client for 30 sec - exiting 12:03:49 (4627): No heartbeat from core client for 30 sec - exiting 12:03:50 (4627): No heartbeat from core client for 30 sec - exiting 12:03:51 (4627): No heartbeat from core client for 30 sec - exiting 12:11:31 (4646): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:11 (4667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:45 (4691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:26:04 (4697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:53 (4742): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:55 (4742): No heartbeat from core client for 30 sec - exiting 14:19:56 (4742): No heartbeat from core client for 30 sec - exiting 14:19:57 (4742): No heartbeat from core client for 30 sec - exiting 14:19:58 (4742): No heartbeat from core client for 30 sec - exiting 14:21:06 (5006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:34:24 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:44 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:59 (5166): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:18 (5350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:01 (5390): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:27 (5409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:29 (5409): No heartbeat from core client for 30 sec - exiting 17:15:16 (5422): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:15:18 (5422): No heartbeat from core client for 30 sec - exiting 17:15:19 (5422): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Thu Jun 13 21:34:06 2013 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/0/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90d46000 - 0x90d54fff /usr/lib/system/libdispatch.dylib 0x90db4000 - 0x90dbbfff /usr/lib/system/libsystem_notify.dylib 0x910df000 - 0x910e0fff /usr/lib/system/libremovefile.dylib 0x91475000 - 0x9147cfff /usr/lib/system/libsystem_dnssd.dylib 0x91a70000 - 0x91a73fff /usr/lib/system/libmathCommon.A.dylib 0x92f8c000 - 0x92f94fff /usr/lib/system/libcopyfile.dylib 0x947ce000 - 0x947cffff /usr/lib/system/libsystem_sandbox.dylib 0x947d0000 - 0x9489bfff /usr/lib/system/libsystem_c.dylib 0x95719000 - 0x9571efff /usr/lib/system/libmacho.dylib 0x961da000 - 0x961dafff /usr/lib/system/libdnsinfo.dylib 0x96661000 - 0x96662fff /usr/lib/system/libquarantine.dylib 0x9753e000 - 0x97546fff /usr/lib/system/liblaunch.dylib 0x98350000 - 0x9837ffff /usr/lib/system/libsystem_info.dylib 0x9853b000 - 0x9853ffff /usr/lib/system/libsystem_network.dylib 0x98819000 - 0x9885cfff /usr/lib/system/libcommonCrypto.dylib 0x98f33000 - 0x98f61fff /usr/lib/libSystem.B.dylib 0x996ea000 - 0x996f2fff /usr/lib/system/libunwind.dylib 0x996f8000 - 0x9975afff /usr/lib/libstdc++.6.dylib 0x997d8000 - 0x997d9fff /usr/lib/system/libsystem_blocks.dylib 0x99bf1000 - 0x99c07fff /usr/lib/system/libxpc.dylib 0x99c08000 - 0x99c16fff /usr/lib/libz.1.dylib 0x9ae21000 - 0x9ae22fff /usr/lib/system/libunc.dylib 0x9ae8a000 - 0x9ae8afff /usr/lib/system/libkeymgr.dylib 0x9b2c6000 - 0x9b2c8fff /usr/lib/system/libdyld.dylib 0x9b5ae000 - 0x9b5b2fff /usr/lib/system/libcache.dylib 0x9c045000 - 0x9c04efff /usr/lib/libc++abi.dylib 0x9c1a2000 - 0x9c1c0fff /usr/lib/system/libsystem_kernel.dylib 0x9c21d000 - 0x9c220fff /usr/lib/system/libcompiler_rt.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jun 2013 19:38:28	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	259,200	229,990	0.8873
13 Jun 2013 10:39:29	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	233,280	207,070	0.8876
13 Jun 2013 01:56:14	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	207,360	184,275	0.8887
12 Jun 2013 16:53:27	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	181,440	161,469	0.8899
12 Jun 2013 08:15:46	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	155,520	138,652	0.8915
12 Jun 2013 00:01:48	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	129,600	114,897	0.8866
11 Jun 2013 16:06:44	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	103,680	90,963	0.8773
11 Jun 2013 07:07:33	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	77,760	67,261	0.8650
10 Jun 2013 22:18:10	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	51,840	43,298	0.8352
10 Jun 2013 15:20:38	1242102	15821220	hadcm3n_o1pf_1980_40_008385818_0	25,920	22,373	0.8632