Task 13538981

Name	hadcm3n_ygru_1900_40_007516652_1
Workunit	7714127
Created	28 Oct 2011, 12:48:17 UTC
Sent	22 Nov 2011, 18:33:39 UTC
Report deadline	22 Feb 2012, 2:00:50 UTC
Received	23 Jan 2012, 15:08:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	810201
Run time	20 days 18 hours 6 min 16 sec
CPU time	20 days 4 hours 25 min 28 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	2.61 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:13:11 (4122): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:31:59 (17606): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:12:23 (4131): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:53:17 (4153): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:45 (4199): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:20 (4210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:21 (4249): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:23 (4265): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:11 (4273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:58 (4311): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:43 (4330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:51 (4366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:54 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:30 (4414): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:41 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:25 (4436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:18 (4448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:44:02 (4465): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:44:40 (4475): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:56 (4485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:57 (4485): No heartbeat from core client for 30 sec - exiting 16:06:58 (4485): No heartbeat from core client for 30 sec - exiting 16:06:59 (4485): No heartbeat from core client for 30 sec - exiting 16:07:00 (4485): No heartbeat from core client for 30 sec - exiting 16:07:01 (4485): No heartbeat from core client for 30 sec - exiting 16:07:02 (4485): No heartbeat from core client for 30 sec - exiting 16:07:03 (4485): No heartbeat from core client for 30 sec - exiting 16:07:04 (4485): No heartbeat from core client for 30 sec - exiting 16:07:05 (4485): No heartbeat from core client for 30 sec - exiting 16:07:06 (4485): No heartbeat from core client for 30 sec - exiting 16:07:07 (4485): No heartbeat from core client for 30 sec - exiting 16:07:08 (4485): No heartbeat from core client for 30 sec - exiting 16:07:09 (4485): No heartbeat from core client for 30 sec - exiting 16:07:10 (4485): No heartbeat from core client for 30 sec - exiting 16:07:11 (4485): No heartbeat from core client for 30 sec - exiting 16:07:12 (4485): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf777b400] [0xf777b430] /lib32/libc.so.6(gsignal+0x51)[0xf7611a21] /lib32/libc.so.6(abort+0x182)[0xf7614e42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75fdc76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a6400] [0xf77a6430] /lib32/libc.so.6(gsignal+0x51)[0xf763ca21] /lib32/libc.so.6(abort+0x182)[0xf763fe42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7628c76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7740400] [0xf7740430] /lib32/libc.so.6(gsignal+0x51)[0xf75d6a21] /lib32/libc.so.6(abort+0x182)[0xf75d9e42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75c2c76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf777d400] [0xf777d430] /lib32/libc.so.6(gsignal+0x51)[0xf7613a21] /lib32/libc.so.6(abort+0x182)[0xf7616e42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75ffc76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7743400] [0xf7743430] /lib32/libc.so.6(gsignal+0x51)[0xf75d9a21] /lib32/libc.so.6(abort+0x182)[0xf75dce42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75c5c76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7756400] [0xf7756430] /lib32/libc.so.6(gsignal+0x51)[0xf75eca21] /lib32/libc.so.6(abort+0x182)[0xf75efe42] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75d8c76] /DATA/BOINC/Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4501, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Jan 2012 06:30:05	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	829,440	1,703,357	2.0536
20 Jan 2012 00:53:01	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	803,520	1,653,680	2.0580
18 Jan 2012 21:29:59	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	777,600	1,604,351	2.0632
17 Jan 2012 17:05:22	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	751,680	1,554,578	2.0681
16 Jan 2012 14:11:26	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	725,760	1,505,137	2.0739
15 Jan 2012 09:40:39	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	699,840	1,455,100	2.0792
14 Jan 2012 06:16:39	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	673,920	1,404,515	2.0841
09 Dec 2011 06:11:59	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	648,000	1,350,281	2.0838
08 Dec 2011 14:49:41	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	622,080	1,295,229	2.0821
07 Dec 2011 23:29:58	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	596,160	1,240,419	2.0807
07 Dec 2011 08:10:58	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	570,240	1,185,511	2.0790
06 Dec 2011 16:57:34	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	544,320	1,131,126	2.0781
06 Dec 2011 01:42:29	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	518,400	1,076,491	2.0766
05 Dec 2011 10:31:24	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	492,480	1,021,987	2.0752
04 Dec 2011 19:06:12	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	466,560	966,783	2.0722
04 Dec 2011 03:38:45	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	440,640	911,620	2.0689
03 Dec 2011 12:22:04	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	414,720	856,781	2.0659
02 Dec 2011 19:46:51	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	388,800	803,251	2.0660
02 Dec 2011 03:50:36	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	362,880	748,904	2.0638
01 Dec 2011 12:32:22	810201	13538981	hadcm3n_ygru_1900_40_007516652_1	336,960	694,313	2.0605