climateprediction.net home page
Task 13396476

Task 13396476

Name hadcm3n_t1nf_1940_40_007445701_3
Workunit 7643204
Created 18 Sep 2011, 19:21:57 UTC
Sent 18 Sep 2011, 19:41:44 UTC
Report deadline 19 Dec 2011, 3:08:55 UTC
Received 24 Sep 2011, 3:10:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1123498
Run time 4 days 18 hours 40 min 39 sec
CPU time 3 days 11 hours 41 min 14 sec
Validate state Invalid
Credit 2,799.36
Device peak FLOPS 2.96 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450E2C  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F7596C0E  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5561, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450E2C  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F7631C0E  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5561, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450E2C  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F7544C0E  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5561, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on devicModel crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08190c00 ***
======= Backtrace: =========
/lib/libc.so.6(+0x6dfbb)[0xf751cfbb]
/lib/libc.so.6(+0x6ed3b)[0xf751dd3b]
/lib/libc.so.6(cfree+0x6d)[0xf7521a9d]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xf7711511]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xf771156d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/libc.so.6(__libc_start_main+0xfe)[0xf74c5c0e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 fd:01 15269                              /opt/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rwxp 0009b000 fd:01 15269                              /opt/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-081a0000 rwxp 00000000 00:00 0                                  [heap]
f6f00000-f6f21000 rwxp 00000000 00:00 0 
f6f21000-f7000000 ---p 00000000 00:00 0 
f7032000-f74ad000 rwxs 00000000 fd:01 16487                              /opt/BOINC/slots/1/135960
f74ad000-f74af000 rwxp 00000000 00:00 0 
f74af000-f7613000 r-xp 00000000 fd:01 393229                             /lib/libc-2.11.2.so
f7613000-f7615000 r-xp 00163000 fd:01 393229                             /lib/libc-2.11.2.so
f7615000-f7616000 rwxp 00165000 fd:01 393229                             /lib/libc-2.11.2.so
f7616000-f7619000 rwxp 00000000 00:00 0 
f7619000-f7635000 r-xp 00000000 fd:01 413115                             /lib/libgcc_s.so.1
f7635000-f7636000 r-xp 0001b000 fd:01 413115                             /lib/libgcc_s.so.1
f7636000-f7637000 rwxp 0001c000 fd:01 413115                             /lib/libgcc_s.so.1
f7637000-f765f000 r-xp 00000000 fd:01 393434                             /lib/libm-2.11.2.so
f765f000-f7660000 r-xp 00027000 fd:01 393434                             /lib/libm-2.11.2.so
f7660000-f7661000 rwxp 00028000 fd:01 393434                             /lib/libm-2.11.2.so
f7661000-f7662000 rwxp 00000000 00:00 0 
f7662000-f7746000 r-xp 00000000 fd:01 543432                             /usr/lib/libstdc++.so.6.0.14
f7746000-f774a000 r-xp 000e3000 fd:01 543432                             /usr/lib/libstdc++.so.6.0.14
f774a000-f774b000 rwxp 000e7000 fd:01 543432                             /usr/lib/libstdc++.so.6.0.14
f774b000-f7752000 rwxp 00000000 00:00 0 
f7752000-f7755000 r-xp 00000000 fd:01 393450                             /lib/libdl-2.11.2.so
f7755000-f7756000 r-xp 00002000 fd:01 393450                             /lib/libdl-2.11.2.so
f7756000-f7757000 rwxp 00003000 fd:01 393450                             /lib/libdl-2.11.2.so
f7757000-f776e000 r-xp 00000000 fd:01 393473                             /lib/libpthread-2.11.2.so
f776e000-f776f000 r-xp 00016000 fd:01 393473                             /lib/libpthread-2.11.2.so
f776f000-f7770000 rwxp 00017000 fd:01 393473                             /lib/libpthread-2.11.2.so
f7770000-f7772000 rwxp 00000000 00:00 0 
f778c000-f778d000 rwxp 00000000 00:00 0 
f778d000-f778e000 ---p 00000000 00:00 0 
f778e000-f7791000 rwxp 00000000 00:00 0 
f7791000-f7793000 rwxs 00000000 fd:01 1225                               /opt/BOINC/slots/1/boinc_mmap_file
f7793000-f77b1000 r-xp 00000000 fd:01 393306                             /lib/ld-2.11.2.so
f77b1000-f77b2000 rwxp 00000000 00:00 0 
f77b2000-f77b3000 r-xp 0001e000 fd:01 393306                             /lib/ld-2.11.2.so
f77b3000-f77b4000 rwxp 0001f000 fd:01 393306                             /lib/ld-2.11.2.so
ffe00000-ffe70000 rw-p 00000000 00:00 0                                  [stack]
ffffe000-fffff000 r-xp 00000000 00:00 0                                  [vdso]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xffffe400]
[0xffffe430]
/lib/libc.so.6(gsignal+0x4f)[0xf74da7af]
/lib/libc.so.6(abort+0x180)[0xf74dc0f0]
/lib/libc.so.6(+0x67f97)[0xf7516f97]
/lib/libc.so.6(+0x6dfbb)[0xf751cfbb]
/lib/libc.so.6(+0x6ed3b)[0xf751dd3b]
/lib/libc.so.6(cfree+0x6d)[0xf7521a9d]
/usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xf7711511]
/usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xf771156d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/libc.so.6(__libc_start_main+0xfe)[0xf74c5c0e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Sep 2011 14:10:04 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 233,280 259,574 1.1127
23 Sep 2011 02:05:23 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 207,360 217,057 1.0468
22 Sep 2011 16:28:19 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 181,440 292,602 1.6127
22 Sep 2011 06:11:05 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 155,520 256,656 1.6503
21 Sep 2011 20:08:32 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 129,600 220,713 1.7030
21 Sep 2011 07:30:38 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 103,680 177,231 1.7094
20 Sep 2011 13:00:50 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 77,760 127,001 1.6332
19 Sep 2011 17:48:41 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 51,840 75,885 1.4638
19 Sep 2011 05:46:03 1123498 13396476 hadcm3n_t1nf_1940_40_007445701_3 25,920 35,948 1.3869


©2024 climateprediction.net