climateprediction.net home page
Task 13632177

Task 13632177

Name hadcm3n_006n_1940_40_007545293_1
Workunit 7742525
Created 10 Nov 2011, 9:30:35 UTC
Sent 15 Nov 2011, 17:13:56 UTC
Report deadline 15 Feb 2012, 0:41:07 UTC
Received 26 Dec 2011, 0:06:08 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1140527
Run time 18 days 10 hours 42 min 25 sec
CPU time 16 days 17 hours 24 min 22 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 2.82 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
16:55:13 (3936): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:55:16 (3936): No heartbeat from core client for 30 sec - exiting
16:55:17 (3936): No heartbeat from core client for 30 sec - exiting
16:55:18 (3936): No heartbeat from core client for 30 sec - exiting
16:55:19 (3936): No heartbeat from core client for 30 sec - exiting
16:55:20 (3936): No heartbeat from core client for 30 sec - exiting
16:55:21 (3936): No heartbeat from core client for 30 sec - exiting
16:55:22 (3936): No heartbeat from core client for 30 sec - exiting
16:55:23 (3936): No heartbeat from core client for 30 sec - exiting
16:55:24 (3936): No heartbeat from core client for 30 sec - exiting
16:55:25 (3936): No heartbeat from core client for 30 sec - exiting
16:55:26 (3936): No heartbeat from core client for 30 sec - exiting
16:55:27 (3936): No heartbeat from core client for 30 sec - exiting
16:55:28 (3936): No heartbeat from core client for 30 sec - exiting
16:55:29 (3936): No heartbeat from core client for 30 sec - exiting
16:55:30 (3936): No heartbeat from core client for 30 sec - exiting
16:55:31 (3936): No heartbeat from core client for 30 sec - exiting
16:55:32 (3936): No heartbeat from core client for 30 sec - exiting
16:55:33 (3936): No heartbeat from core client for 30 sec - exiting
16:55:34 (3936): No heartbeat from core client for 30 sec - exiting
16:55:35 (3936): No heartbeat from core client for 30 sec - exiting
16:55:36 (3936): No heartbeat from core client for 30 sec - exiting
16:55:37 (3936): No heartbeat from core client for 30 sec - exiting
16:55:38 (3936): No heartbeat from core client for 30 sec - exiting
16:55:39 (3936): No heartbeat from core client for 30 sec - exiting
16:55:40 (3936): No heartbeat from core client for 30 sec - exiting
16:55:41 (3936): No heartbeat from core client for 30 sec - exiting
16:55:42 (3936): No heartbeat from core client for 30 sec - exiting
16:55:43 (3936): No heartbeat from core client for 30 sec - exiting
16:55:44 (3936): No heartbeat from core client for 30 sec - exiting
16:55:45 (3936): No heartbeat from core client for 30 sec - exiting
16:55:46 (3936): No heartbeat from core client for 30 sec - exiting
16:55:47 (3936): No heartbeat from core client for 30 sec - exiting
16:55:48 (3936): No heartbeat from core client for 30 sec - exiting
16:55:49 (3936): No heartbeat from core client for 30 sec - exiting
16:55:50 (3936): No heartbeat from core client for 30 sec - exiting
16:55:51 (3936): No heartbeat from core client for 30 sec - exiting
16:55:52 (3936): No heartbeat from core client for 30 sec - exiting
16:55:53 (3936): No heartbeat from core client for 30 sec - exiting
16:55:54 (3936): No heartbeat from core client for 30 sec - exiting
16:55:55 (3936): No heartbeat from core client for 30 sec - exiting
16:55:56 (3936): No heartbeat from core client for 30 sec - exiting
16:55:57 (3936): No heartbeat from core client for 30 sec - exiting
16:55:58 (3936): No heartbeat from core client for 30 sec - exiting
16:55:59 (3936): No heartbeat from core client for 30 sec - exiting
16:56:00 (3936): No heartbeat from core client for 30 sec - exiting
16:56:01 (3936): No heartbeat from core client for 30 sec - exiting
16:56:02 (3936): No heartbeat from core client for 30 sec - exiting
16:56:03 (3936): No heartbeat from core client for 30 sec - exiting
16:56:04 (3936): No heartbeat from core client for 30 sec - exiting
16:56:05 (3936): No heartbeat from core client for 30 sec - exiting
16:56:06 (3936): No heartbeat from core client for 30 sec - exiting
16:56:07 (3936): No heartbeat from core client for 30 sec - exiting
16:56:08 (3936): No heartbeat from core client for 30 sec - exiting
16:56:09 (3936): No heartbeat from core client for 30 sec - exiting
16:56:10 (3936): No heartbeat from core client for 30 sec - exiting
16:56:11 (3936): No heartbeat from core client for 30 sec - exiting
16:56:12 (3936): No heartbeat from core client for 30 sec - exiting
16:56:13 (3936): No heartbeat from core client for 30 sec - exiting
16:56:14 (3936): No heartbeat from core client for 30 sec - exiting
16:56:15 (3936): No heartbeat from core client for 30 sec - exiting
16:56:16 (3936): No heartbeat from core client for 30 sec - exiting
16:56:17 (3936): No heartbeat from core client for 30 sec - exiting
16:56:18 (3936): No heartbeat from core client for 30 sec - exiting
16:56:19 (3936): No heartbeat from core client for 30 sec - exiting
16:56:20 (3936): No heartbeat from core client for 30 sec - exiting
16:56:21 (3936): No heartbeat from core client for 30 sec - exiting
16:56:22 (3936): No heartbeat from core client for 30 sec - exiting
16:56:23 (3936): No heartbeat from core client for 30 sec - exiting
16:56:24 (3936): No heartbeat from core client for 30 sec - exiting
16:56:25 (3936): No heartbeat from core client for 30 sec - exiting
16:56:26 (3936): No heartbeat from core client for 30 sec - exiting
16:56:27 (3936): No heartbeat from core client for 30 sec - exiting
16:56:28 (3936): No heartbeat from core client for 30 sec - exiting
16:56:29 (3936): No heartbeat from core client for 30 sec - exiting
16:56:30 (3936): No heartbeat from core client for 30 sec - exiting
16:56:31 (3936): No heartbeat from core client for 30 sec - exiting
16:56:32 (3936): No heartbeat from core client for 30 sec - exiting
16:56:33 (3936): No heartbeat from core client for 30 sec - exiting
16:56:34 (3936): No heartbeat from core client for 30 sec - exiting
16:56:35 (3936): No heartbeat from core client for 30 sec - exiting
16:56:36 (3936): No heartbeat from core client for 30 sec - exiting
16:56:37 (3936): No heartbeat from core client for 30 sec - exiting
16:56:38 (3936): No heartbeat from core client for 30 sec - exiting
16:56:39 (3936): No heartbeat from core client for 30 sec - exiting
16:56:40 (3936): No heartbeat from core client for 30 sec - exiting
16:56:41 (3936): No heartbeat from core client for 30 sec - exiting
16:56:42 (3936): No heartbeat from core client for 30 sec - exiting
16:56:43 (3936): No heartbeat from core client for 30 sec - exiting
16:56:44 (3936): No heartbeat from core client for 30 sec - exiting
16:56:45 (3936): No heartbeat from core client for 30 sec - exiting
16:56:46 (3936): No heartbeat from core client for 30 sec - exiting
16:56:47 (3936): No heartbeat from core client for 30 sec - exiting
16:56:48 (3936): No heartbeat from core client for 30 sec - exiting
16:56:49 (3936): No heartbeat from core client for 30 sec - exiting
16:56:50 (3936): No heartbeat from core client for 30 sec - exiting
16:56:51 (3936): No heartbeat from core client for 30 sec - exiting
16:56:52 (3936): No heartbeat from core client for 30 sec - exiting
16:56:53 (3936): No heartbeat from core client for 30 sec - exiting
16:56:54 (3936): No heartbeat from core client for 30 sec - exiting
16:56:55 (3936): No heartbeat from core client for 30 sec - exiting
16:56:56 (3936): No heartbeat from core client for 30 sec - exiting
16:56:57 (3936): No heartbeat from core client for 30 sec - exiting
16:56:58 (3936): No heartbeat from core client for 30 sec - exiting
16:56:59 (3936): No heartbeat from core client for 30 sec - exiting
16:57:00 (3936): No heartbeat from core client for 30 sec - exiting
16:57:01 (3936): No heartbeat from core client for 30 sec - exiting
16:57:02 (3936): No heartbeat from core client for 30 sec - exiting
16:57:03 (3936): No heartbeat from core client for 30 sec - exiting
16:57:04 (3936): No heartbeat from core client for 30 sec - exiting
16:57:05 (3936): No heartbeat from core client for 30 sec - exiting
16:57:06 (3936): No heartbeat from core client for 30 sec - exiting
16:57:07 (3936): No heartbeat from core client for 30 sec - exiting
16:57:08 (3936): No heartbeat from core client for 30 sec - exiting
16:57:09 (3936): No heartbeat from core client for 30 sec - exiting
16:57:10 (3936): No heartbeat from core client for 30 sec - exiting
16:58:24 (7787): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:58:25 (7787): No heartbeat from core client for 30 sec - exiting
16:58:26 (7787): No heartbeat from core client for 30 sec - exiting
17:01:31 (7921): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:01:32 (7921): No heartbeat from core client for 30 sec - exiting
17:01:33 (7921): No heartbeat from core client for 30 sec - exiting
17:01:34 (7921): No heartbeat from core client for 30 sec - exiting
20:25:43 (4350): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:25:44 (4350): No heartbeat from core client for 30 sec - exiting
20:25:45 (4350): No heartbeat from core client for 30 sec - exiting
20:25:46 (4350): No heartbeat from core client for 30 sec - exiting
20:25:47 (4350): No heartbeat from core client for 30 sec - exiting
20:25:48 (4350): No heartbeat from core client for 30 sec - exiting
20:25:49 (4350): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
20:27:07 (6660): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:27:08 (6660): No heartbeat from core client for 30 sec - exiting
20:27:09 (6660): No heartbeat from core client for 30 sec - exiting
20:27:10 (6660): No heartbeat from core client for 30 sec - exiting
20:27:11 (6660): No heartbeat from core client for 30 sec - exiting
20:27:12 (6660): No heartbeat from core client for 30 sec - exiting
20:27:13 (6660): No heartbeat from core client for 30 sec - exiting
20:27:14 (6660): No heartbeat from core client for 30 sec - exiting
20:27:15 (6660): No heartbeat from core client for 30 sec - exiting
20:27:16 (6660): No heartbeat from core client for 30 sec - exiting
20:27:17 (6660): No heartbeat from core client for 30 sec - exiting
20:27:18 (6660): No heartbeat from core client for 30 sec - exiting
20:27:19 (6660): No heartbeat from core client for 30 sec - exiting
20:27:20 (6660): No heartbeat from core client for 30 sec - exiting
20:27:21 (6660): No heartbeat from core client for 30 sec - exiting
20:27:22 (6660): No heartbeat from core client for 30 sec - exiting
20:27:23 (6660): No heartbeat from core client for 30 sec - exiting
20:27:24 (6660): No heartbeat from core client for 30 sec - exiting
20:27:25 (6660): No heartbeat from core client for 30 sec - exiting
20:27:26 (6660): No heartbeat from core client for 30 sec - exiting
20:27:27 (6660): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08bf0b10 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6c231)[0xf7493231]
/lib32/libc.so.6(+0x6dab8)[0xf7494ab8]
/lib32/libc.so.6(cfree+0x6d)[0xf7497b9d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7681a91]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7681aed]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf743dbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 fc:08 13109854                           /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rwxp 0009b000 fc:08 13109854                           /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rwxp 00000000 00:00 0 
08b9b000-08c00000 rwxp 00000000 00:00 0                                  [heap]
f6e00000-f6e21000 rwxp 00000000 00:00 0 
f6e21000-f6f00000 ---p 00000000 00:00 0 
f6faa000-f7425000 rwxs 00000000 fc:08 13107720                           /igel2/Vol1/BOINC/slots/22/131830
f7425000-f7427000 rwxp 00000000 00:00 0 
f7427000-f757a000 r-xp 00000000 08:32 4063237                            /lib32/libc-2.11.1.so
f757a000-f757b000 ---p 00153000 08:32 4063237                            /lib32/libc-2.11.1.so
f757b000-f757d000 r-xp 00153000 08:32 4063237                            /lib32/libc-2.11.1.so
f757d000-f757e000 rwxp 00155000 08:32 4063237                            /lib32/libc-2.11.1.so
f757e000-f7581000 rwxp 00000000 00:00 0 
f7581000-f759e000 r-xp 00000000 08:32 3282217                            /usr/lib32/libgcc_s.so.1
f759e000-f759f000 r-xp 0001c000 08:32 3282217                            /usr/lib32/libgcc_s.so.1
f759f000-f75a0000 rwxp 0001d000 08:32 3282217                            /usr/lib32/libgcc_s.so.1
f75a0000-f75c4000 r-xp 00000000 08:32 4063241                            /lib32/libm-2.11.1.so
f75c4000-f75c5000 r-xp 00023000 08:32 4063241                            /lib32/libm-2.11.1.so
f75c5000-f75c6000 rwxp 00024000 08:32 4063241                            /lib32/libm-2.11.1.so
f75c6000-f76af000 r-xp 00000000 08:32 3282231                            /usr/lib32/libstdc++.so.6.0.13
f76af000-f76b0000 ---p 000e9000 08:32 3282231                            /usr/lib32/libstdc++.so.6.0.13
f76b0000-f76b4000 r-xp 000e9000 08:32 3282231                            /usr/lib32/libstdc++.so.6.0.13
f76b4000-f76b5000 rwxp 000ed000 08:32 3282231                            /usr/lib32/libstdc++.so.6.0.13
f76b5000-f76bc000 rwxp 00000000 00:00 0 
f76bc000-f76be000 r-xp 00000000 08:32 4063240                            /lib32/libdl-2.11.1.so
f76be000-f76bf000 r-xp 00001000 08:32 4063240                            /lib32/libdl-2.11.1.so
f76bf000-f76c0000 rwxp 00002000 08:32 4063240                            /lib32/libdl-2.11.1.so
f76c0000-f76d5000 r-xp 00000000 08:32 4063251                            /lib32/libpthread-2.11.1.so
f76d5000-f76d6000 r-xp 00014000 08:32 4063251                            /lib32/libpthread-2.11.1.so
f76d6000-f76d7000 rwxp 00015000 08:32 4063251                            /lib32/libpthread-2.11.1.so
f76d7000-f76da000 rwxp 00000000 00:00 0 
f76ee000-f76ef000 rwxp 00000000 00:00 0 
f76ef000-f76f0000 ---p 00000000 00:00 0 
f76f0000-f76f3000 rwxp 00000000 00:00 0 
f76f3000-f76f5000 rwxs 00000000 fc:08 13107733                           /igel2/Vol1/BOINC/slots/22/boinc_mmap_file
f76f5000-f76f7000 rwxp 00000000 00:00 0 
f76f7000-f76f8000 r-xp 00000000 00:00 0                                  [vdso]
f76f8000-f7714000 r-xp 00000000 08:32 4063234                            /lib32/ld-2.11.1.so
f7714000-f7715000 r-xp 0001b000 08:32 4063234                            /lib32/ld-2.11.1.so
f7715000-f7716000 rwxp 0001c000 08:32 4063234                            /lib32/ld-2.11.1.so
ff814000-ff884000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf76f7400]
[0xf76f7430]
/lib32/libc.so.6(gsignal+0x51)[0xf7451921]
/lib32/libc.so.6(abort+0x182)[0xf7454d52]
/lib32/libc.so.6(+0x6213d)[0xf748913d]
/lib32/libc.so.6(+0x6c231)[0xf7493231]
/lib32/libc.so.6(+0x6dab8)[0xf7494ab8]
/lib32/libc.so.6(cfree+0x6d)[0xf7497b9d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7681a91]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7681aed]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf743dbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Dec 2011 00:08:27 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 777,600 1,445,060 1.8584
25 Dec 2011 08:48:58 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 751,680 1,395,186 1.8561
24 Dec 2011 22:11:04 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 725,760 1,354,914 1.8669
24 Dec 2011 05:54:26 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 699,840 1,300,992 1.8590
21 Dec 2011 04:41:58 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 673,920 1,246,744 1.8500
20 Dec 2011 08:47:50 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 648,000 1,194,987 1.8441
19 Dec 2011 19:11:58 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 622,080 1,141,858 1.8355
16 Dec 2011 04:55:33 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 596,160 1,101,035 1.8469
15 Dec 2011 15:52:43 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 570,240 1,053,388 1.8473
15 Dec 2011 02:07:34 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 544,320 1,002,602 1.8419
14 Dec 2011 11:38:36 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 518,400 950,635 1.8338
13 Dec 2011 21:09:29 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 492,480 898,772 1.8250
11 Dec 2011 20:29:31 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 466,560 845,181 1.8115
11 Dec 2011 04:30:45 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 440,640 791,198 1.7956
10 Dec 2011 11:31:40 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 414,720 735,710 1.7740
28 Nov 2011 00:42:56 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 388,800 685,917 1.7642
27 Nov 2011 09:20:52 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 362,880 636,728 1.7547
26 Nov 2011 19:05:25 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 336,960 587,604 1.7438
25 Nov 2011 22:31:45 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 311,040 537,149 1.7269
24 Nov 2011 10:01:55 1140527 13632177 hadcm3n_006n_1940_40_007545293_1 285,120 489,636 1.7173


©2024 climateprediction.net