Task 13412529

Name	hadcm3n_u5xh_1980_40_007460657_1
Workunit	7658160
Created	23 Sep 2011, 0:33:23 UTC
Sent	23 Sep 2011, 23:25:06 UTC
Report deadline	24 Dec 2011, 6:52:17 UTC
Received	17 Nov 2011, 3:07:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1151942
Run time	10 days 20 hours 59 min 27 sec
CPU time	7 days 4 hours 45 min 55 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.23 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 03:11:16 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:21 (4484): No heartbeat from core client for 30 sec - exiting 03:16:22 (5816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:37 (5816): No heartbeat from core client for 30 sec - exiting 03:16:38 (5816): No heartbeat from core client for 30 sec - exiting 03:16:39 (5816): No heartbeat from core client for 30 sec - exiting 03:16:40 (5816): No heartbeat from core client for 30 sec - exiting 03:16:41 (5816): No heartbeat from core client for 30 sec - exiting 03:21:15 (5949): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:32 (6045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:39 (820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:57 (11861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:36:31 (13468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:56 (13534): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:19 (24025): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:20 (24025): No heartbeat from core client for 30 sec - exiting 02:16:21 (24025): No heartbeat from core client for 30 sec - exiting 02:16:22 (24025): No heartbeat from core client for 30 sec - exiting 02:16:23 (24025): No heartbeat from core client for 30 sec - exiting 02:16:24 (24025): No heartbeat from core client for 30 sec - exiting 02:19:37 (25404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:38 (25404): No heartbeat from core client for 30 sec - exiting 02:19:39 (25404): No heartbeat from core client for 30 sec - exiting 02:47:20 (26064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:21 (26064): No heartbeat from core client for 30 sec - exiting 02:47:22 (26064): No heartbeat from core client for 30 sec - exiting 02:47:23 (26064): No heartbeat from core client for 30 sec - exiting 02:47:24 (26064): No heartbeat from core client for 30 sec - exiting 02:47:25 (26064): No heartbeat from core client for 30 sec - exiting 02:47:26 (26064): No heartbeat from core client for 30 sec - exiting 03:03:26 (29154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:32 (31332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:33 (31332): No heartbeat from core client for 30 sec - exiting 03:45:34 (31332): No heartbeat from core client for 30 sec - exiting 03:45:35 (31332): No heartbeat from core client for 30 sec - exiting 03:45:36 (31332): No heartbeat from core client for 30 sec - exiting 03:45:37 (31332): No heartbeat from core client for 30 sec - exiting 03:45:38 (31332): No heartbeat from core client for 30 sec - exiting 03:45:39 (31332): No heartbeat from core client for 30 sec - exiting 03:45:40 (31332): No heartbeat from core client for 30 sec - exiting 03:45:41 (31332): No heartbeat from core client for 30 sec - exiting 03:45:42 (31332): No heartbeat from core client for 30 sec - exiting 03:45:43 (31332): No heartbeat from core client for 30 sec - exiting 06:05:10 (4595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 08:51:07 (21841): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:21:03 (20138): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:22 (30310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:38 (30486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:36 (21060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:37 (8430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:07:18 (11840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:09 (7499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:53 (8564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:04 (9591): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:15 (9794): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:16 (9794): No heartbeat from core client for 30 sec - exiting 13:37:17 (9794): No heartbeat from core client for 30 sec - exiting 13:37:18 (9794): No heartbeat from core client for 30 sec - exiting 13:41:24 (14038): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:57 (14926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:50 (15139): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:51 (15139): No heartbeat from core client for 30 sec - exiting 13:54:52 (15139): No heartbeat from core client for 30 sec - exiting 13:54:53 (15139): No heartbeat from core client for 30 sec - exiting 13:54:54 (15139): No heartbeat from core client for 30 sec - exiting 13:54:55 (15139): No heartbeat from core client for 30 sec - exiting 13:54:56 (15139): No heartbeat from core client for 30 sec - exiting 13:57:56 (18189): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:08 (19362): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:04 (20398): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:18:05 (20398): No heartbeat from core client for 30 sec - exiting 14:18:06 (20398): No heartbeat from core client for 30 sec - exiting 14:18:07 (20398): No heartbeat from core client for 30 sec - exiting 14:18:08 (20398): No heartbeat from core client for 30 sec - exiting 14:18:09 (20398): No heartbeat from core client for 30 sec - exiting 14:18:10 (20398): No heartbeat from core client for 30 sec - exiting 14:18:11 (20398): No heartbeat from core client for 30 sec - exiting 14:18:12 (20398): No heartbeat from core client for 30 sec - exiting 14:18:13 (20398): No heartbeat from core client for 30 sec - exiting 14:18:15 (20398): No heartbeat from core client for 30 sec - exiting 14:18:16 (20398): No heartbeat from core client for 30 sec - exiting 14:18:17 (20398): No heartbeat from core client for 30 sec - exiting 14:18:18 (20398): No heartbeat from core client for 30 sec - exiting 14:18:19 (20398): No heartbeat from core client for 30 sec - exiting 14:18:20 (20398): No heartbeat from core client for 30 sec - exiting 14:18:21 (20398): No heartbeat from core client for 30 sec - exiting 14:18:22 (20398): No heartbeat from core client for 30 sec - exiting 14:18:23 (20398): No heartbeat from core client for 30 sec - exiting 14:18:24 (20398): No heartbeat from core client for 30 sec - exiting 14:18:25 (20398): No heartbeat from core client for 30 sec - exiting 14:18:26 (20398): No heartbeat from core client for 30 sec - exiting 14:18:27 (20398): No heartbeat from core client for 30 sec - exiting 14:18:28 (20398): No heartbeat from core client for 30 sec - exiting 14:18:29 (20398): No heartbeat from core client for 30 sec - exiting 14:18:30 (20398): No heartbeat from core client for 30 sec - exiting 14:18:31 (20398): No heartbeat from core client for 30 sec - exiting 14:58:34 (22820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:35 (22820): No heartbeat from core client for 30 sec - exiting 14:58:36 (22820): No heartbeat from core client for 30 sec - exiting 14:58:37 (22820): No heartbeat from core client for 30 sec - exiting 14:58:38 (22820): No heartbeat from core client for 30 sec - exiting 14:58:39 (22820): No heartbeat from core client for 30 sec - exiting 14:58:40 (22820): No heartbeat from core client for 30 sec - exiting 14:58:41 (22820): No heartbeat from core client for 30 sec - exiting 15:02:06 (32029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:51 (32267): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:42:11 (32693): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:42:12 (32693): No heartbeat from core client for 30 sec - exiting 15:42:13 (32693): No heartbeat from core client for 30 sec - exiting 15:42:14 (32693): No heartbeat from core client for 30 sec - exiting 15:42:15 (32693): No heartbeat from core client for 30 sec - exiting 15:42:16 (32693): No heartbeat from core client for 30 sec - exiting 15:42:17 (32693): No heartbeat from core client for 30 sec - exiting 15:42:18 (32693): No heartbeat from core client for 30 sec - exiting 15:42:19 (32693): No heartbeat from core client for 30 sec - exiting 15:42:20 (32693): No heartbeat from core client for 30 sec - exiting 15:42:21 (32693): No heartbeat from core client for 30 sec - exiting 15:44:27 (7593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:38 (8584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:39 (8584): No heartbeat from core client for 30 sec - exiting 16:24:20 (15961): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:35 (17036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:21 (5813): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:22 (5813): No heartbeat from core client for 30 sec - exiting 19:29:20 (21629): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:41 (22600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:43 (22600): No heartbeat from core client for 30 sec - exiting 20:10:44 (22600): No heartbeat from core client for 30 sec - exiting 20:10:45 (22600): No heartbeat from core client for 30 sec - exiting 20:10:46 (22600): No heartbeat from core client for 30 sec - exiting 03:06:42 (30879): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:04:09 (28225): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:06 (14121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:35:25 (16865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:26 (2329): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:58:34 (22901): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:58:36 (22901): No heartbeat from core client for 30 sec - exiting 13:58:37 (22901): No heartbeat from core client for 30 sec - exiting 13:58:38 (22901): No heartbeat from core client for 30 sec - exiting 13:58:39 (22901): No heartbeat from core client for 30 sec - exiting 18:24:06 (19868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:07 (19868): No heartbeat from core client for 30 sec - exiting 18:27:18 (9937): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:19 (9937): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x087f0a28 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6c231)[0xf74d0231] /lib32/libc.so.6(+0x6dab8)[0xf74d1ab8] /lib32/libc.so.6(cfree+0x6d)[0xf74d4b9d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76bea91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76beaed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf747abd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 fb:00 44331351 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 fb:00 44331351 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0879b000-08800000 rw-p 00000000 00:00 0 [heap] f6e00000-f6e21000 rw-p 00000000 00:00 0 f6e21000-f6f00000 ---p 00000000 00:00 0 f6fe7000-f7462000 rw-s 00000000 fb:00 44313077 /var/lib/boinc-client/slots/13/138125 f7462000-f7464000 rw-p 00000000 00:00 0 f7464000-f75b7000 r-xp 00000000 fb:00 213647365 /lib32/libc-2.11.1.so f75b7000-f75b8000 ---p 00153000 fb:00 213647365 /lib32/libc-2.11.1.so f75b8000-f75ba000 r--p 00153000 fb:00 213647365 /lib32/libc-2.11.1.so f75ba000-f75bb000 rw-p 00155000 fb:00 213647365 /lib32/libc-2.11.1.so f75bb000-f75be000 rw-p 00000000 00:00 0 f75be000-f75db000 r-xp 00000000 fb:00 109451375 /usr/lib32/libgcc_s.so.1 f75db000-f75dc000 r--p 0001c000 fb:00 109451375 /usr/lib32/libgcc_s.so.1 f75dc000-f75dd000 rw-p 0001d000 fb:00 109451375 /usr/lib32/libgcc_s.so.1 f75dd000-f7601000 r-xp 00000000 fb:00 213647369 /lib32/libm-2.11.1.so f7601000-f7602000 r--p 00023000 fb:00 213647369 /lib32/libm-2.11.1.so f7602000-f7603000 rw-p 00024000 fb:00 213647369 /lib32/libm-2.11.1.so f7603000-f76ec000 r-xp 00000000 fb:00 109451378 /usr/lib32/libstdc++.so.6.0.13 f76ec000-f76ed000 ---p 000e9000 fb:00 109451378 /usr/lib32/libstdc++.so.6.0.13 f76ed000-f76f1000 r--p 000e9000 fb:00 109451378 /usr/lib32/libstdc++.so.6.0.13 f76f1000-f76f2000 rw-p 000ed000 fb:00 109451378 /usr/lib32/libstdc++.so.6.0.13 f76f2000-f76f9000 rw-p 00000000 00:00 0 f76f9000-f76fb000 r-xp 00000000 fb:00 213647368 /lib32/libdl-2.11.1.so f76fb000-f76fc000 r--p 00001000 fb:00 213647368 /lib32/libdl-2.11.1.so f76fc000-f76fd000 rw-p 00002000 fb:00 213647368 /lib32/libdl-2.11.1.so f76fd000-f76fe000 rw-p 00000000 00:00 0 f76fe000-f7713000 r-xp 00000000 fb:00 213647379 /lib32/libpthread-2.11.1.so f7713000-f7714000 r--p 00014000 fb:00 213647379 /lib32/libpthread-2.11.1.so f7714000-f7715000 rw-p 00015000 fb:00 213647379 /lib32/libpthread-2.11.1.so f7715000-f7717000 rw-p 00000000 00:00 0 f7723000-f7724000 rw-p 00000000 00:00 0 f7724000-f7725000 ---p 00000000 00:00 0 f7725000-f7728000 rw-p 00000000 00:00 0 f7728000-f772a000 rw-s 00000000 fb:00 44311518 /var/lib/boinc-client/slots/13/boinc_mmap_file f772a000-f772c000 rw-p 00000000 00:00 0 f772c000-f772d000 r-xp 00000000 00:00 0 [vdso] f772d000-f7749000 r-xp 00000000 fb:00 213647362 /lib32/ld-2.11.1.so f7749000-f774a000 r--p 0001b000 fb:00 213647362 /lib32/ld-2.11.1.so f774a000-f774b000 rw-p 0001c000 fb:00 213647362 /lib32/ld-2.11.1.so ff977000-ff9e7000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf772c400] [0xf772c430] /lib32/libc.so.6(gsignal+0x51)[0xf748e921] /lib32/libc.so.6(abort+0x182)[0xf7491d52] /lib32/libc.so.6(+0x6213d)[0xf74c613d] /lib32/libc.so.6(+0x6c231)[0xf74d0231] /lib32/libc.so.6(+0x6dab8)[0xf74d1ab8] /lib32/libc.so.6(cfree+0x6d)[0xf74d4b9d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76bea91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76beaed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf747abd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Nov 2011 03:11:54	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	259,200	621,954	2.3995
15 Nov 2011 18:07:50	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	233,280	561,965	2.4090
15 Nov 2011 18:08:00	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	207,360	501,616	2.4191
15 Nov 2011 18:07:55	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	181,440	440,629	2.4285
15 Nov 2011 18:08:01	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	155,520	379,136	2.4379
10 Nov 2011 05:40:23	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	129,600	314,532	2.4269
09 Nov 2011 10:21:37	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	103,680	250,105	2.4123
26 Sep 2011 11:21:24	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	77,760	195,207	2.5104
25 Sep 2011 15:55:04	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	51,840	129,696	2.5019
24 Sep 2011 20:04:51	1151942	13412529	hadcm3n_u5xh_1980_40_007460657_1	25,920	64,340	2.4823