climateprediction.net home page
Task 13605615

Task 13605615

Name hadcm3n_yjoe_1980_40_007538094_0
Workunit 7735326
Created 5 Nov 2011, 19:00:18 UTC
Sent 7 Nov 2011, 10:48:36 UTC
Report deadline 6 Feb 2012, 18:15:47 UTC
Received 9 Dec 2011, 9:13:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 942439
Run time 27 days 5 hours 51 min 17 sec
CPU time 11 days 18 hours 37 min 32 sec
Validate state Invalid
Credit 12,441.60
Device peak FLOPS 1.51 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
01:42:38 (20923): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:42:43 (20923): No heartbeat from core client for 30 sec - exiting
01:42:44 (20923): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    

SETPOS: Seek Failed: No space left on device
SETPOS: Unit 22 to Word Address 325632 Failed with Error Code -1

Model crashed: SETPOS: Unit 22 to Word Address 325632 Failed with Error Code -1

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
04:06:51 (2795): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09aa1268 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x591591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x592de8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x595ecd]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x1cb741]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x1cb79d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x53cbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
00110000-001f9000 r-xp 00000000 08:01 26715180   /usr/lib/libstdc++.so.6.0.13
001f9000-001fa000 ---p 000e9000 08:01 26715180   /usr/lib/libstdc++.so.6.0.13
001fa000-001fe000 r--p 000e9000 08:01 26715180   /usr/lib/libstdc++.so.6.0.13
001fe000-001ff000 rw-p 000ed000 08:01 26715180   /usr/lib/libstdc++.so.6.0.13
001ff000-00206000 rw-p 00000000 00:00 0 
00299000-0029a000 r-xp 00000000 00:00 0          [vdso]
003d0000-003f4000 r-xp 00000000 08:01 10092560   /lib/tls/i686/cmov/libm-2.11.1.so
003f4000-003f5000 r--p 00023000 08:01 10092560   /lib/tls/i686/cmov/libm-2.11.1.so
003f5000-003f6000 rw-p 00024000 08:01 10092560   /lib/tls/i686/cmov/libm-2.11.1.so
00522000-00524000 r-xp 00000000 08:01 10092549   /lib/tls/i686/cmov/libdl-2.11.1.so
00524000-00525000 r--p 00001000 08:01 10092549   /lib/tls/i686/cmov/libdl-2.11.1.so
00525000-00526000 rw-p 00002000 08:01 10092549   /lib/tls/i686/cmov/libdl-2.11.1.so
00526000-00679000 r-xp 00000000 08:01 10092556   /lib/tls/i686/cmov/libc-2.11.1.so
00679000-0067a000 ---p 00153000 08:01 10092556   /lib/tls/i686/cmov/libc-2.11.1.so
0067a000-0067c000 r--p 00153000 08:01 10092556   /lib/tls/i686/cmov/libc-2.11.1.so
0067c000-0067d000 rw-p 00155000 08:01 10092556   /lib/tls/i686/cmov/libc-2.11.1.so
0067d000-00680000 rw-p 00000000 00:00 0 
00c41000-00c5c000 r-xp 00000000 08:01 27208321   /lib/ld-2.11.1.so
00c5c000-00c5d000 r--p 0001a000 08:01 27208321   /lib/ld-2.11.1.so
00c5d000-00c5e000 rw-p 0001b000 08:01 27208321   /lib/ld-2.11.1.so
00d53000-00d68000 r-xp 00000000 08:01 10092551   /lib/tls/i686/cmov/libpthread-2.11.1.so
00d68000-00d69000 r--p 00014000 08:01 10092551   /lib/tls/i686/cmov/libpthread-2.11.1.so
00d69000-00d6a000 rw-p 00015000 08:01 10092551   /lib/tls/i686/cmov/libpthread-2.11.1.so
00d6a000-00d6c000 rw-p 00000000 00:00 0 
00fc9000-00fe6000 r-xp 00000000 08:01 27197571   /lib/libgcc_s.so.1
00fe6000-00fe7000 r--p 0001c000 08:01 27197571   /lib/libgcc_s.so.1
00fe7000-00fe8000 rw-p 0001d000 08:01 27197571   /lib/libgcc_s.so.1
08048000-080e3000 r-xp 00000000 08:01 393889     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 08:01 393889     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
09a4b000-09ab1000 rw-p 00000000 00:00 0          [heap]
b7200000-b7221000 rw-p 00000000 00:00 0 
b7221000-b7300000 ---p 00000000 00:00 0 
b73d0000-b784b000 rw-s 00000000 08:01 11542565   /var/lib/boinc-client/slots/5/140155
b784b000-b784e000 rw-p 00000000 00:00 0 
b7862000-b7863000 rw-p 00000000 00:00 0 
b7863000-b7864000 ---p 00000000 00:00 0 
b7864000-b7867000 rw-p 00000000 00:00 0 
b7867000-b7869000 rw-s 00000000 08:01 11542562   /var/lib/boinc-client/slots/5/boinc_mmap_file
b7869000-b786b000 rw-p 00000000 00:00 0 
bf8ae000-bf91e000 rw-p 00000000 00:00 0          [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0x299400]
[0x299422]
/lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x550651]
/lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x553a82]
/lib/tls/i686/cmov/libc.so.6(+0x6149d)[0x58749d]
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x591591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x592de8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x595ecd]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x1cb741]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x1cb79d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x53cbd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Dec 2011 09:17:48 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 1,036,800 1,017,445 0.9813
08 Dec 2011 07:02:41 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 1,010,880 958,419 0.9481
07 Dec 2011 08:31:06 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 984,960 897,897 0.9116
06 Dec 2011 11:15:38 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 959,040 840,381 0.8763
04 Dec 2011 15:06:31 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 933,120 2,096,977 2.2473
03 Dec 2011 17:43:47 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 907,200 2,036,702 2.2450
02 Dec 2011 18:24:59 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 881,280 1,976,770 2.2431
01 Dec 2011 16:38:25 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 855,360 1,917,736 2.2420
30 Nov 2011 23:17:20 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 829,440 1,860,367 2.2429
30 Nov 2011 06:51:03 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 803,520 1,801,780 2.2424
29 Nov 2011 10:12:05 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 777,600 1,744,644 2.2436
28 Nov 2011 17:58:07 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 751,680 1,687,056 2.2444
27 Nov 2011 21:26:50 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 725,760 1,627,880 2.2430
27 Nov 2011 02:08:43 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 699,840 1,571,391 2.2454
26 Nov 2011 08:04:23 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 673,920 1,512,438 2.2442
25 Nov 2011 15:32:40 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 648,000 1,453,374 2.2429
24 Nov 2011 23:33:25 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 622,080 1,396,179 2.2444
24 Nov 2011 07:01:07 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 596,160 1,336,925 2.2426
23 Nov 2011 14:48:45 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 570,240 1,278,747 2.2425
22 Nov 2011 22:39:53 942439 13605615 hadcm3n_yjoe_1980_40_007538094_0 544,320 1,221,165 2.2435


©2024 climateprediction.net