climateprediction.net home page
Task 13600590

Task 13600590

Name hadcm3n_o2jr_1940_40_007536091_0
Workunit 7733323
Created 5 Nov 2011, 11:28:41 UTC
Sent 17 Nov 2011, 12:10:30 UTC
Report deadline 16 Feb 2012, 19:37:41 UTC
Received 30 Nov 2011, 6:24:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1167372
Run time 6 days 18 hours 17 min 25 sec
CPU time 6 days 15 hours 7 min 13 sec
Validate state Invalid
Credit 3,110.40
Device peak FLOPS 1.99 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.12.33</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
18:33:49 (24666): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:46:44 (29878): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:19:29 (30777): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
18:48:01 (2993): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:57:55 (3017): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:08:48 (3055): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:24:46 (3102): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:29:23 (3164): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:37:05 (4221): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:37:42 (4249): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:53:32 (4259): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
17:28:56 (6469): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
01:37:37 (8393): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
03:59:58 (9518): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:02:21 (9535): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:03:57 (9549): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:04:43 (9562): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:39:35 (9580): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
06:40:57 (10042): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
06:41:43 (10056): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
07:03:17 (10067): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:03:59 (10115): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:05:25 (10125): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:08:13 (10137): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:24:38 (10604): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:44:06 (10662): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:44:59 (10730): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
11:20:02 (11178): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:20:57 (11208): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:22:28 (11224): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:23:45 (11237): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:26:38 (11253): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:01:15 (11276): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:02:07 (11377): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:07:05 (11390): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:14:45 (11415): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
06:24:00 (13637): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x094d04a8 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb74aabc2]
/lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb74ab862]
/lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb74ae94d]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb76ad80f]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb76ad86b]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb7455113]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 fc:03 795628     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 fc:03 795628     /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
0947b000-094e0000 rw-p 00000000 00:00 0          [heap]
b6e00000-b6e21000 rw-p 00000000 00:00 0 
b6e21000-b6f00000 ---p 00000000 00:00 0 
b6fbe000-b7439000 rw-s 00000000 fc:03 1705963    /var/lib/boinc-client/slots/5/136705
b7439000-b743c000 rw-p 00000000 00:00 0 
b743c000-b75b2000 r-xp 00000000 fc:03 9965631    /lib/i386-linux-gnu/libc-2.13.so
b75b2000-b75b4000 r--p 00176000 fc:03 9965631    /lib/i386-linux-gnu/libc-2.13.so
b75b4000-b75b5000 rw-p 00178000 fc:03 9965631    /lib/i386-linux-gnu/libc-2.13.so
b75b5000-b75b8000 rw-p 00000000 00:00 0 
b75b8000-b75d4000 r-xp 00000000 fc:03 9965627    /lib/i386-linux-gnu/libgcc_s.so.1
b75d4000-b75d5000 r--p 0001b000 fc:03 9965627    /lib/i386-linux-gnu/libgcc_s.so.1
b75d5000-b75d6000 rw-p 0001c000 fc:03 9965627    /lib/i386-linux-gnu/libgcc_s.so.1
b75d6000-b75fe000 r-xp 00000000 fc:03 9965635    /lib/i386-linux-gnu/libm-2.13.so
b75fe000-b75ff000 r--p 00028000 fc:03 9965635    /lib/i386-linux-gnu/libm-2.13.so
b75ff000-b7600000 rw-p 00029000 fc:03 9965635    /lib/i386-linux-gnu/libm-2.13.so
b7600000-b76de000 r-xp 00000000 fc:03 15729125   /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
b76de000-b76df000 ---p 000de000 fc:03 15729125   /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
b76df000-b76e3000 r--p 000de000 fc:03 15729125   /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
b76e3000-b76e4000 rw-p 000e2000 fc:03 15729125   /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
b76e4000-b76ec000 rw-p 00000000 00:00 0 
b76ec000-b76ef000 r-xp 00000000 fc:03 9965634    /lib/i386-linux-gnu/libdl-2.13.so
b76ef000-b76f0000 r--p 00002000 fc:03 9965634    /lib/i386-linux-gnu/libdl-2.13.so
b76f0000-b76f1000 rw-p 00003000 fc:03 9965634    /lib/i386-linux-gnu/libdl-2.13.so
b76f1000-b7708000 r-xp 00000000 fc:03 9965645    /lib/i386-linux-gnu/libpthread-2.13.so
b7708000-b7709000 r--p 00016000 fc:03 9965645    /lib/i386-linux-gnu/libpthread-2.13.so
b7709000-b770a000 rw-p 00017000 fc:03 9965645    /lib/i386-linux-gnu/libpthread-2.13.so
b770a000-b770c000 rw-p 00000000 00:00 0 
b771b000-b771c000 rw-p 00000000 00:00 0 
b771c000-b771d000 ---p 00000000 00:00 0 
b771d000-b7720000 rw-p 00000000 00:00 0 
b7720000-b7722000 rw-s 00000000 fc:03 1705960    /var/lib/boinc-client/slots/5/boinc_mmap_file
b7722000-b7724000 rw-p 00000000 00:00 0 
b7724000-b7725000 r-xp 00000000 00:00 0          [vdso]
b7725000-b7743000 r-xp 00000000 fc:03 9965628    /lib/i386-linux-gnu/ld-2.13.so
b7743000-b7744000 r--p 0001d000 fc:03 9965628    /lib/i386-linux-gnu/ld-2.13.so
b7744000-b7745000 rw-p 0001e000 fc:03 9965628    /lib/i386-linux-gnu/ld-2.13.so
bfd82000-bfdf1000 rw-p 00000000 00:00 0          [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xb7724400]
[0xb7724424]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb7469c8f]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb746d2b5]
/lib/i386-linux-gnu/libc.so.6(+0x63dfc)[0xb749fdfc]
/lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb74aabc2]
/lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb74ab862]
/lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb74ae94d]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb76ad80f]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb76ad86b]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb7455113]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
29 Nov 2011 23:48:46 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 259,200 572,823 2.2100
28 Nov 2011 17:22:55 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 233,280 516,554 2.2143
27 Nov 2011 00:33:13 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 207,360 460,138 2.2190
22 Nov 2011 09:54:04 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 181,440 403,192 2.2222
21 Nov 2011 17:05:43 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 155,520 344,643 2.2161
20 Nov 2011 22:45:57 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 129,600 287,342 2.2171
20 Nov 2011 06:12:17 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 103,680 230,045 2.2188
19 Nov 2011 13:35:05 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 77,760 171,522 2.2058
18 Nov 2011 20:59:56 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 51,840 113,894 2.1970
18 Nov 2011 04:33:05 1167372 13600590 hadcm3n_o2jr_1940_40_007536091_0 25,920 56,843 2.1930


©2024 climateprediction.net