Task 17481956

Name	hadcm3n_x127_1940_40_009148642_1
Workunit	9278978
Created	25 Nov 2014, 6:52:03 UTC
Sent	25 Nov 2014, 6:52:19 UTC
Report deadline	24 Feb 2015, 14:19:30 UTC
Received	4 Dec 2014, 6:26:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1292656
Run time	8 days 22 hours 40 min 29 sec
CPU time	8 days 8 hours 40 min 55 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	4.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:32:53 (3102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:05:58 (3675): No heartbeat from core client for 30 sec - exiting 08:05:59 (3675): No heartbeat from core client for 30 sec - exiting 08:06:00 (3675): No heartbeat from core client for 30 sec - exiting 08:06:01 (3675): No heartbeat from core client for 30 sec - exiting 08:06:02 (3675): No heartbeat from core client for 30 sec - exiting 08:06:03 (3675): No heartbeat from core client for 30 sec - exiting 08:06:04 (3675): No heartbeat from core client for 30 sec - exiting 08:06:05 (3675): No heartbeat from core client for 30 sec - exiting 08:06:06 (3675): No heartbeat from core client for 30 sec - exiting 08:06:07 (3675): No heartbeat from core client for 30 sec - exiting 08:06:08 (3675): No heartbeat from core client for 30 sec - exiting 08:06:09 (3675): No heartbeat from core client for 30 sec - exiting 08:06:10 (3675): No heartbeat from core client for 30 sec - exiting 08:06:11 (3675): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:53:33 (4661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:53:34 (4661): No heartbeat from core client for 30 sec - exiting 09:53:35 (4661): No heartbeat from core client for 30 sec - exiting 09:53:36 (4661): No heartbeat from core client for 30 sec - exiting 09:53:37 (4661): No heartbeat from core client for 30 sec - exiting 09:53:38 (4661): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:05:46 (7056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:53 (7056): No heartbeat from core client for 30 sec - exiting 10:05:54 (7056): No heartbeat from core client for 30 sec - exiting 10:05:55 (7056): No heartbeat from core client for 30 sec - exiting 10:05:56 (7056): No heartbeat from core client for 30 sec - exiting 10:05:57 (7056): No heartbeat from core client for 30 sec - exiting 10:05:58 (7056): No heartbeat from core client for 30 sec - exiting 10:05:59 (7056): No heartbeat from core client for 30 sec - exiting 10:06:00 (7056): No heartbeat from core client for 30 sec - exiting 10:06:01 (7056): No heartbeat from core client for 30 sec - exiting 10:06:02 (7056): No heartbeat from core client for 30 sec - exiting 10:06:03 (7056): No heartbeat from core client for 30 sec - exiting 10:06:04 (7056): No heartbeat from core client for 30 sec - exiting 10:06:05 (7056): No heartbeat from core client for 30 sec - exiting 10:06:06 (7056): No heartbeat from core client for 30 sec - exiting 10:06:07 (7056): No heartbeat from core client for 30 sec - exiting 10:06:08 (7056): No heartbeat from core client for 30 sec - exiting 10:06:09 (7056): No heartbeat from core client for 30 sec - exiting 10:06:10 (7056): No heartbeat from core client for 30 sec - exiting 10:06:11 (7056): No heartbeat from core client for 30 sec - exiting 10:06:12 (7056): No heartbeat from core client for 30 sec - exiting 10:06:13 (7056): No heartbeat from core client for 30 sec - exiting 10:06:14 (7056): No heartbeat from core client for 30 sec - exiting 10:06:15 (7056): No heartbeat from core client for 30 sec - exiting 10:06:16 (7056): No heartbeat from core client for 30 sec - exiting 10:06:17 (7056): No heartbeat from core client for 30 sec - exiting 10:06:18 (7056): No heartbeat from core client for 30 sec - exiting 10:06:19 (7056): No heartbeat from core client for 30 sec - exiting 10:06:20 (7056): No heartbeat from core client for 30 sec - exiting 10:06:21 (7056): No heartbeat from core client for 30 sec - exiting 10:06:22 (7056): No heartbeat from core client for 30 sec - exiting 10:06:23 (7056): No heartbeat from core client for 30 sec - exiting 10:06:24 (7056): No heartbeat from core client for 30 sec - exiting 10:06:25 (7056): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:16:19 (7313): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:20 (7313): No heartbeat from core client for 30 sec - exiting 08:16:21 (7313): No heartbeat from core client for 30 sec - exiting 08:16:22 (7313): No heartbeat from core client for 30 sec - exiting 08:16:23 (7313): No heartbeat from core client for 30 sec - exiting 08:16:24 (7313): No heartbeat from core client for 30 sec - exiting 08:16:25 (7313): No heartbeat from core client for 30 sec - exiting 08:16:26 (7313): No heartbeat from core client for 30 sec - exiting 08:16:27 (7313): No heartbeat from core client for 30 sec - exiting 08:16:28 (7313): No heartbeat from core client for 30 sec - exiting 08:16:29 (7313): No heartbeat from core client for 30 sec - exiting 08:16:30 (7313): No heartbeat from core client for 30 sec - exiting 08:16:31 (7313): No heartbeat from core client for 30 sec - exiting 08:16:32 (7313): No heartbeat from core client for 30 sec - exiting 08:16:33 (7313): No heartbeat from core client for 30 sec - exiting 08:16:34 (7313): No heartbeat from core client for 30 sec - exiting 08:16:35 (7313): No heartbeat from core client for 30 sec - exiting 08:16:36 (7313): No heartbeat from core client for 30 sec - exiting 08:16:37 (7313): No heartbeat from core client for 30 sec - exiting 08:16:38 (7313): No heartbeat from core client for 30 sec - exiting 08:16:39 (7313): No heartbeat from core client for 30 sec - exiting 08:16:40 (7313): No heartbeat from core client for 30 sec - exiting 08:16:41 (7313): No heartbeat from core client for 30 sec - exiting 08:16:42 (7313): No heartbeat from core client for 30 sec - exiting 08:16:43 (7313): No heartbeat from core client for 30 sec - exiting 08:16:44 (7313): No heartbeat from core client for 30 sec - exiting 08:16:45 (7313): No heartbeat from core client for 30 sec - exiting 08:16:46 (7313): No heartbeat from core client for 30 sec - exiting 08:16:47 (7313): No heartbeat from core client for 30 sec - exiting 08:16:48 (7313): No heartbeat from core client for 30 sec - exiting 08:16:49 (7313): No heartbeat from core client for 30 sec - exiting 08:16:50 (7313): No heartbeat from core client for 30 sec - exiting 08:16:51 (7313): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf773c400] [0xf773c430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7531e0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7535455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf751d4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7754400] [0xf7754430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7549e0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf754d455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75354d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf770a400] [0xf770a430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74ffe0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7503455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74eb4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7711400] [0xf7711430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7506e0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf750a455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74f24d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f1400] [0xf76f1430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74e6e0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf74ea455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74d24d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f4400] [0xf76f4430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74e9e0f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf74ed455] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74d54d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4710, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Dec 2014 05:10:18	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	803,520	719,809	0.8958
03 Dec 2014 21:04:05	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	777,600	696,506	0.8957
03 Dec 2014 12:57:49	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	751,680	673,191	0.8956
03 Dec 2014 03:55:41	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	725,760	649,556	0.8950
02 Dec 2014 19:51:17	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	699,840	626,267	0.8949
02 Dec 2014 12:03:21	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	673,920	602,789	0.8945
02 Dec 2014 04:57:04	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	648,000	578,971	0.8935
01 Dec 2014 22:17:09	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	622,080	555,883	0.8936
01 Dec 2014 16:11:08	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	596,160	532,888	0.8939
01 Dec 2014 08:56:15	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	570,240	509,305	0.8931
01 Dec 2014 02:18:11	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	544,320	486,053	0.8930
30 Nov 2014 19:39:40	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	518,400	463,100	0.8933
30 Nov 2014 13:07:17	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	492,480	440,132	0.8937
30 Nov 2014 06:35:24	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	466,560	417,185	0.8942
30 Nov 2014 00:00:45	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	440,640	394,233	0.8947
29 Nov 2014 17:24:00	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	414,720	371,260	0.8952
29 Nov 2014 10:52:54	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	388,800	348,279	0.8958
29 Nov 2014 04:53:04	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	362,880	325,300	0.8964
28 Nov 2014 21:41:10	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	336,960	302,230	0.8969
28 Nov 2014 15:36:27	1292656	17481956	hadcm3n_x127_1940_40_009148642_1	311,040	279,285	0.8979