climateprediction.net home page
Task 13416054

Task 13416054

Name hadcm3n_u068_1980_40_007460766_2
Workunit 7658269
Created 23 Sep 2011, 22:14:15 UTC
Sent 23 Sep 2011, 22:24:17 UTC
Report deadline 24 Dec 2011, 5:51:28 UTC
Received 15 Nov 2011, 17:33:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1151942
Run time 5 days 9 hours 24 min 15 sec
CPU time 4 days 16 hours 56 min 33 sec
Validate state Invalid
Credit 1,866.24
Device peak FLOPS 2.23 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
03:11:15 (19909): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:11:21 (19909): No heartbeat from core client for 30 sec - exiting
03:16:22 (5812): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:21:15 (5944): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:27:33 (6041): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:58:36 (816): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:58:38 (816): No heartbeat from core client for 30 sec - exiting
00:34:59 (11857): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:36:30 (13462): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:59:55 (13530): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:16:19 (24021): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:16:20 (24021): No heartbeat from core client for 30 sec - exiting
02:16:21 (24021): No heartbeat from core client for 30 sec - exiting
02:16:22 (24021): No heartbeat from core client for 30 sec - exiting
02:16:23 (24021): No heartbeat from core client for 30 sec - exiting
02:16:24 (24021): No heartbeat from core client for 30 sec - exiting
02:19:37 (25402): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:47:19 (26060): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:47:20 (26060): No heartbeat from core client for 30 sec - exiting
02:47:21 (26060): No heartbeat from core client for 30 sec - exiting
02:47:22 (26060): No heartbeat from core client for 30 sec - exiting
02:47:23 (26060): No heartbeat from core client for 30 sec - exiting
02:47:24 (26060): No heartbeat from core client for 30 sec - exiting
02:47:25 (26060): No heartbeat from core client for 30 sec - exiting
03:02:35 (29150): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:02:36 (29150): No heartbeat from core client for 30 sec - exiting
03:02:37 (29150): No heartbeat from core client for 30 sec - exiting
03:02:38 (29150): No heartbeat from core client for 30 sec - exiting
03:02:39 (29150): No heartbeat from core client for 30 sec - exiting
03:02:40 (29150): No heartbeat from core client for 30 sec - exiting
03:02:41 (29150): No heartbeat from core client for 30 sec - exiting
03:45:32 (31328): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:05:11 (4591): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITHEAD: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
08:51:07 (21836): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
03:21:04 (20137): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:23:20 (30306): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:38:37 (30482): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:38:38 (30482): No heartbeat from core client for 30 sec - exiting
03:12:36 (21056): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:12:39 (21056): No heartbeat from core client for 30 sec - exiting
03:30:37 (8413): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:07:19 (11836): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:11:06 (7495): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:11:07 (7495): No heartbeat from core client for 30 sec - exiting
13:11:08 (7495): No heartbeat from core client for 30 sec - exiting
13:11:09 (7495): No heartbeat from core client for 30 sec - exiting
13:11:10 (7495): No heartbeat from core client for 30 sec - exiting
13:11:11 (7495): No heartbeat from core client for 30 sec - exiting
13:11:12 (7495): No heartbeat from core client for 30 sec - exiting
13:13:33 (8446): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:18:04 (9587): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:37:15 (9778): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:37:16 (9778): No heartbeat from core client for 30 sec - exiting
13:37:17 (9778): No heartbeat from core client for 30 sec - exiting
13:37:18 (9778): No heartbeat from core client for 30 sec - exiting
13:39:09 (13924): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:41:56 (14815): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:54:49 (15136): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:54:51 (15136): No heartbeat from core client for 30 sec - exiting
13:54:52 (15136): No heartbeat from core client for 30 sec - exiting
13:54:53 (15136): No heartbeat from core client for 30 sec - exiting
13:54:54 (15136): No heartbeat from core client for 30 sec - exiting
13:54:55 (15136): No heartbeat from core client for 30 sec - exiting
13:54:56 (15136): No heartbeat from core client for 30 sec - exiting
13:57:56 (18187): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:03:06 (19358): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:18:02 (20391): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:18:03 (20391): No heartbeat from core client for 30 sec - exiting
14:18:04 (20391): No heartbeat from core client for 30 sec - exiting
14:18:05 (20391): No heartbeat from core client for 30 sec - exiting
14:18:06 (20391): No heartbeat from core client for 30 sec - exiting
14:18:07 (20391): No heartbeat from core client for 30 sec - exiting
14:18:08 (20391): No heartbeat from core client for 30 sec - exiting
14:18:09 (20391): No heartbeat from core client for 30 sec - exiting
14:18:10 (20391): No heartbeat from core client for 30 sec - exiting
14:18:11 (20391): No heartbeat from core client for 30 sec - exiting
14:18:12 (20391): No heartbeat from core client for 30 sec - exiting
14:18:13 (20391): No heartbeat from core client for 30 sec - exiting
14:18:14 (20391): No heartbeat from core client for 30 sec - exiting
14:18:15 (20391): No heartbeat from core client for 30 sec - exiting
14:18:16 (20391): No heartbeat from core client for 30 sec - exiting
14:18:17 (20391): No heartbeat from core client for 30 sec - exiting
14:18:18 (20391): No heartbeat from core client for 30 sec - exiting
14:18:19 (20391): No heartbeat from core client for 30 sec - exiting
14:18:20 (20391): No heartbeat from core client for 30 sec - exiting
14:18:21 (20391): No heartbeat from core client for 30 sec - exiting
14:18:22 (20391): No heartbeat from core client for 30 sec - exiting
14:18:23 (20391): No heartbeat from core client for 30 sec - exiting
14:18:24 (20391): No heartbeat from core client for 30 sec - exiting
14:18:25 (20391): No heartbeat from core client for 30 sec - exiting
14:18:26 (20391): No heartbeat from core client for 30 sec - exiting
14:18:27 (20391): No heartbeat from core client for 30 sec - exiting
14:18:28 (20391): No heartbeat from core client for 30 sec - exiting
14:18:29 (20391): No heartbeat from core client for 30 sec - exiting
14:18:30 (20391): No heartbeat from core client for 30 sec - exiting
14:18:31 (20391): No heartbeat from core client for 30 sec - exiting
14:58:34 (22818): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:58:36 (22818): No heartbeat from core client for 30 sec - exiting
14:58:37 (22818): No heartbeat from core client for 30 sec - exiting
14:58:38 (22818): No heartbeat from core client for 30 sec - exiting
14:58:39 (22818): No heartbeat from core client for 30 sec - exiting
14:58:40 (22818): No heartbeat from core client for 30 sec - exiting
14:58:41 (22818): No heartbeat from core client for 30 sec - exiting
14:58:42 (22818): No heartbeat from core client for 30 sec - exiting
15:00:46 (31866): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:04:50 (32263): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:42:11 (32688): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:42:12 (32688): No heartbeat from core client for 30 sec - exiting
15:42:13 (32688): No heartbeat from core client for 30 sec - exiting
15:42:14 (32688): No heartbeat from core client for 30 sec - exiting
15:42:15 (32688): No heartbeat from core client for 30 sec - exiting
15:42:16 (32688): No heartbeat from core client for 30 sec - exiting
15:42:17 (32688): No heartbeat from core client for 30 sec - exiting
15:42:18 (32688): No heartbeat from core client for 30 sec - exiting
15:42:19 (32688): No heartbeat from core client for 30 sec - exiting
15:42:20 (32688): No heartbeat from core client for 30 sec - exiting
15:42:21 (32688): No heartbeat from core client for 30 sec - exiting
15:42:22 (32688): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7731400]
[0xf7731430]
/lib32/libc.so.6(gsignal+0x51)[0xf75a8921]
/lib32/libc.so.6(abort+0x182)[0xf75abd52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf7594bd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7713400]
[0xf7713430]
/lib32/libc.so.6(gsignal+0x51)[0xf758a921]
/lib32/libc.so.6(abort+0x182)[0xf758dd52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf7576bd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf779a400]
[0xf779a430]
/lib32/libc.so.6(gsignal+0x51)[0xf7611921]
/lib32/libc.so.6(abort+0x182)[0xf7614d52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf75fdbd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7785400]
[0xf7785430]
/lib32/libc.so.6(gsignal+0x51)[0xf75fc921]
/lib32/libc.so.6(abort+0x182)[0xf75ffd52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf75e8bd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7722400]
[0xf7722430]
/lib32/libc.so.6(gsignal+0x51)[0xf7599921]
/lib32/libc.so.6(abort+0x182)[0xf759cd52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf7585bd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (10 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7728400]
[0xf7728430]
/lib32/libc.so.6(gsignal+0x51)[0xf759f921]
/lib32/libc.so.6(abort+0x182)[0xf75a2d52]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf758bbd6]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7591, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Nov 2011 18:07:59 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 155,520 377,849 2.4296
10 Nov 2011 03:38:24 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 129,600 313,747 2.4209
09 Nov 2011 08:13:02 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 103,680 249,437 2.4058
26 Sep 2011 09:04:57 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 77,760 194,773 2.5048
25 Sep 2011 14:29:09 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 51,840 130,101 2.5097
24 Sep 2011 18:48:46 1151942 13416054 hadcm3n_u068_1980_40_007460766_2 25,920 64,277 2.4798


©2024 climateprediction.net