climateprediction.net home page
Task 12985057

Task 12985057

Name hadcm3n_i7qe_1899_40_007298075_0
Workunit 7495499
Created 17 Jun 2011, 11:38:57 UTC
Sent 17 Jun 2011, 11:39:04 UTC
Report deadline 16 Sep 2011, 19:06:15 UTC
Received 16 Jul 2011, 6:32:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1149451
Run time 25 days 18 hours 37 min 6 sec
CPU time 24 days 23 hours 1 min 13 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 1.62 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
CPDN Monitor - Quit request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crasheCPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08b57cd0 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x534591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x535de8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x538ecd]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x414741]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x41479d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x4dfbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
00110000-00112000 r-xp 00000000 08:01 269625     /lib/tls/i686/cmov/libdl-2.11.1.so
00112000-00113000 r--p 00001000 08:01 269625     /lib/tls/i686/cmov/libdl-2.11.1.so
00113000-00114000 rw-p 00002000 08:01 269625     /lib/tls/i686/cmov/libdl-2.11.1.so
001ec000-00209000 r-xp 00000000 08:01 260692     /lib/libgcc_s.so.1
00209000-0020a000 r--p 0001c000 08:01 260692     /lib/libgcc_s.so.1
0020a000-0020b000 rw-p 0001d000 08:01 260692     /lib/libgcc_s.so.1
00359000-00442000 r-xp 00000000 08:01 136624     /usr/lib/libstdc++.so.6.0.13
00442000-00443000 ---p 000e9000 08:01 136624     /usr/lib/libstdc++.so.6.0.13
00443000-00447000 r--p 000e9000 08:01 136624     /usr/lib/libstdc++.so.6.0.13
00447000-00448000 rw-p 000ed000 08:01 136624     /usr/lib/libstdc++.so.6.0.13
00448000-0044f000 rw-p 00000000 00:00 0 
004a3000-004c7000 r-xp 00000000 08:01 269626     /lib/tls/i686/cmov/libm-2.11.1.so
004c7000-004c8000 r--p 00023000 08:01 269626     /lib/tls/i686/cmov/libm-2.11.1.so
004c8000-004c9000 rw-p 00024000 08:01 269626     /lib/tls/i686/cmov/libm-2.11.1.so
004c9000-0061c000 r-xp 00000000 08:01 269583     /lib/tls/i686/cmov/libc-2.11.1.so
0061c000-0061d000 ---p 00153000 08:01 269583     /lib/tls/i686/cmov/libc-2.11.1.so
0061d000-0061f000 r--p 00153000 08:01 269583     /lib/tls/i686/cmov/libc-2.11.1.so
0061f000-00620000 rw-p 00155000 08:01 269583     /lib/tls/i686/cmov/libc-2.11.1.so
00620000-00623000 rw-p 00000000 00:00 0 
00c44000-00c59000 r-xp 00000000 08:01 269778     /lib/tls/i686/cmov/libpthread-2.11.1.so
00c59000-00c5a000 r--p 00014000 08:01 269778     /lib/tls/i686/cmov/libpthread-2.11.1.so
00c5a000-00c5b000 rw-p 00015000 08:01 269778     /lib/tls/i686/cmov/libpthread-2.11.1.so
00c5b000-00c5d000 rw-p 00000000 00:00 0 
00d57000-00d72000 r-xp 00000000 08:01 264932     /lib/ld-2.11.1.so
00d72000-00d73000 r--p 0001a000 08:01 264932     /lib/ld-2.11.1.so
00d73000-00d74000 rw-p 0001b000 08:01 264932     /lib/ld-2.11.1.so
00e59000-00e5a000 r-xp 00000000 00:00 0          [vdso]
08048000-080e3000 r-xp 00000000 08:01 412759     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 08:01 412759     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
08b02000-08b68000 rw-p 00000000 00:00 0          [heap]
b7200000-b7221000 rw-p 00000000 00:00 0 
b7221000-b7300000 ---p 00000000 00:00 0 
b73e8000-b7863000 rw-s 00000000 08:01 452690     /var/lib/boinc-client/slots/2/138300
b7863000-b7866000 rw-p 00000000 00:00 0 
b7871000-b7872000 rw-p 00000000 00:00 0 
b7872000-b7873000 ---p 00000000 00:00 0 
b7873000-b7876000 rw-p 00000000 00:00 0 
b7876000-b7878000 rw-s 00000000 08:01 423058     /var/lib/boinc-client/slots/2/boinc_mmap_file
b7878000-b787a000 rw-p 00000000 00:00 0 
bf878000-bf8e7000 rw-p 00000000 00:00 0          [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xe59400]
[0xe59422]
/lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x4f3651]
/lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x4f6a82]
/lib/tls/i686/cmov/libc.so.6(+0x6149d)[0x52a49d]
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x534591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x535de8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x538ecd]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x414741]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x41479d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x4dfbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
25 Jul 2011 15:03:26 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 777,600 2,156,463 2.7732
25 Jul 2011 13:38:11 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 751,680 2,087,700 2.7774
25 Jul 2011 13:38:10 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 725,760 2,015,254 2.7767
25 Jul 2011 13:38:11 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 699,840 1,938,231 2.7695
25 Jul 2011 13:38:11 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 673,920 1,868,068 2.7719
11 Jul 2011 03:26:18 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 648,000 1,798,646 2.7757
10 Jul 2011 07:31:16 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 622,080 1,729,056 2.7795
09 Jul 2011 11:46:20 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 596,160 1,659,041 2.7829
08 Jul 2011 14:23:56 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 570,240 1,587,673 2.7842
07 Jul 2011 18:03:45 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 544,320 1,516,107 2.7853
07 Jul 2011 15:36:02 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 518,400 1,444,080 2.7856
05 Jul 2011 20:46:24 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 492,480 1,373,051 2.7880
04 Jul 2011 23:18:39 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 466,560 1,297,277 2.7805
04 Jul 2011 01:43:20 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 440,640 1,221,068 2.7711
03 Jul 2011 02:17:51 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 414,720 1,144,888 2.7606
02 Jul 2011 04:18:05 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 388,800 1,068,711 2.7487
01 Jul 2011 06:27:00 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 362,880 992,801 2.7359
30 Jun 2011 05:53:23 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 336,960 917,021 2.7215
29 Jun 2011 07:27:50 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 311,040 840,771 2.7031
28 Jun 2011 07:08:07 1149451 12985057 hadcm3n_i7qe_1899_40_007298075_0 285,120 768,692 2.6960


©2024 climateprediction.net