Task 13413052

Name	hadcm3n_u4na_1980_40_007460851_1
Workunit	7658354
Created	23 Sep 2011, 1:10:26 UTC
Sent	23 Sep 2011, 18:44:19 UTC
Report deadline	24 Dec 2011, 2:11:30 UTC
Received	17 Nov 2011, 3:07:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1151942
Run time	10 days 23 hours 45 min 48 sec
CPU time	7 days 6 hours 56 min 28 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.23 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 03:11:15 (12141): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:22 (12141): No heartbeat from core client for 30 sec - exiting 03:16:22 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:37 (5804): No heartbeat from core client for 30 sec - exiting 03:16:38 (5804): No heartbeat from core client for 30 sec - exiting 03:16:39 (5804): No heartbeat from core client for 30 sec - exiting 03:16:40 (5804): No heartbeat from core client for 30 sec - exiting 03:16:41 (5804): No heartbeat from core client for 30 sec - exiting 03:21:15 (5936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:32 (6033): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:36 (804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:59 (11849): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:36:31 (13460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:55 (13526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:19 (24019): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:36 (25394): No heartbeat from core client for 30 sec - exiting 02:19:39 (25394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:20 (26011): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:21 (26011): No heartbeat from core client for 30 sec - exiting 02:47:22 (26011): No heartbeat from core client for 30 sec - exiting 02:47:23 (26011): No heartbeat from core client for 30 sec - exiting 02:47:24 (26011): No heartbeat from core client for 30 sec - exiting 02:47:25 (26011): No heartbeat from core client for 30 sec - exiting 02:47:26 (26011): No heartbeat from core client for 30 sec - exiting 03:03:26 (29146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:32 (31322): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:11 (4573): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 08:51:06 (21830): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:21:04 (20135): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:20 (30298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:38 (30469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:36 (21048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:37 (8396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:07:19 (11828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:51 (7377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:32 (8409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:04 (9577): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:15 (9707): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:06 (13886): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:25 (14804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:50 (15126): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:56 (18150): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:06 (19324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:02 (20354): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:03 (20354): No heartbeat from core client for 30 sec - exiting 14:18:04 (20354): No heartbeat from core client for 30 sec - exiting 14:18:05 (20354): No heartbeat from core client for 30 sec - exiting 14:18:06 (20354): No heartbeat from core client for 30 sec - exiting 14:18:07 (20354): No heartbeat from core client for 30 sec - exiting 14:18:08 (20354): No heartbeat from core client for 30 sec - exiting 14:18:09 (20354): No heartbeat from core client for 30 sec - exiting 14:18:10 (20354): No heartbeat from core client for 30 sec - exiting 14:18:11 (20354): No heartbeat from core client for 30 sec - exiting 14:18:12 (20354): No heartbeat from core client for 30 sec - exiting 14:18:13 (20354): No heartbeat from core client for 30 sec - exiting 14:18:14 (20354): No heartbeat from core client for 30 sec - exiting 14:18:15 (20354): No heartbeat from core client for 30 sec - exiting 14:18:16 (20354): No heartbeat from core client for 30 sec - exiting 14:18:17 (20354): No heartbeat from core client for 30 sec - exiting 14:18:18 (20354): No heartbeat from core client for 30 sec - exiting 14:18:19 (20354): No heartbeat from core client for 30 sec - exiting 14:18:20 (20354): No heartbeat from core client for 30 sec - exiting 14:18:21 (20354): No heartbeat from core client for 30 sec - exiting 14:18:22 (20354): No heartbeat from core client for 30 sec - exiting 14:18:23 (20354): No heartbeat from core client for 30 sec - exiting 14:18:24 (20354): No heartbeat from core client for 30 sec - exiting 14:18:25 (20354): No heartbeat from core client for 30 sec - exiting 14:18:26 (20354): No heartbeat from core client for 30 sec - exiting 14:18:27 (20354): No heartbeat from core client for 30 sec - exiting 14:18:28 (20354): No heartbeat from core client for 30 sec - exiting 14:18:29 (20354): No heartbeat from core client for 30 sec - exiting 14:18:30 (20354): No heartbeat from core client for 30 sec - exiting 14:18:31 (20354): No heartbeat from core client for 30 sec - exiting 14:58:35 (22810): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:46 (31198): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:47 (31198): No heartbeat from core client for 30 sec - exiting 15:00:48 (31198): No heartbeat from core client for 30 sec - exiting 15:00:49 (31198): No heartbeat from core client for 30 sec - exiting 15:00:50 (31198): No heartbeat from core client for 30 sec - exiting 15:00:51 (31198): No heartbeat from core client for 30 sec - exiting 15:00:52 (31198): No heartbeat from core client for 30 sec - exiting 15:00:53 (31198): No heartbeat from core client for 30 sec - exiting 15:00:54 (31198): No heartbeat from core client for 30 sec - exiting 15:00:55 (31198): No heartbeat from core client for 30 sec - exiting 15:00:56 (31198): No heartbeat from core client for 30 sec - exiting 15:04:03 (32221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:42:10 (32680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:42:11 (32680): No heartbeat from core client for 30 sec - exiting 15:42:12 (32680): No heartbeat from core client for 30 sec - exiting 15:42:13 (32680): No heartbeat from core client for 30 sec - exiting 15:42:14 (32680): No heartbeat from core client for 30 sec - exiting 15:42:15 (32680): No heartbeat from core client for 30 sec - exiting 15:42:16 (32680): No heartbeat from core client for 30 sec - exiting 15:42:17 (32680): No heartbeat from core client for 30 sec - exiting 15:42:18 (32680): No heartbeat from core client for 30 sec - exiting 15:42:19 (32680): No heartbeat from core client for 30 sec - exiting 15:42:20 (32680): No heartbeat from core client for 30 sec - exiting 15:42:21 (32680): No heartbeat from core client for 30 sec - exiting 15:44:27 (7581): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:39 (8573): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:46 (15955): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:34 (16996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:20 (5783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:21 (5783): No heartbeat from core client for 30 sec - exiting 19:28:34 (21589): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:44 (22579): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:45 (22579): No heartbeat from core client for 30 sec - exiting 20:10:46 (22579): No heartbeat from core client for 30 sec - exiting 20:10:47 (22579): No heartbeat from core client for 30 sec - exiting 03:06:42 (30870): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:58 (30870): No heartbeat from core client for 30 sec - exiting 03:04:09 (28214): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:05 (14111): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:35:25 (16863): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:27 (16863): No heartbeat from core client for 30 sec - exiting 17:55:26 (2310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:06 (22893): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:07 (22893): No heartbeat from core client for 30 sec - exiting 18:27:18 (9915): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:19 (9915): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation Stack trace (15 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf774c400] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x806c0d5] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x806e5f2] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8072509] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8077f47] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80781a3] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e1b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf749abd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Nov 2011 02:11:41	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	259,200	629,782	2.4297
15 Nov 2011 18:08:01	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	233,280	570,248	2.4445
15 Nov 2011 18:07:55	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	207,360	510,512	2.4620
15 Nov 2011 18:07:56	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	181,440	449,061	2.4750
15 Nov 2011 18:07:58	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	155,520	387,376	2.4908
10 Nov 2011 02:31:40	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	129,600	322,861	2.4912
09 Nov 2011 07:12:52	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	103,680	259,123	2.4993
26 Sep 2011 08:04:20	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	77,760	195,418	2.5131
25 Sep 2011 12:27:57	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	51,840	129,164	2.4916
24 Sep 2011 16:47:30	1151942	13413052	hadcm3n_u4na_1980_40_007460851_1	25,920	63,770	2.4603