Task 15451589

Name	hadcm3n_zeqp_1880_40_008250380_1
Workunit	8405504
Created	22 Nov 2012, 5:12:22 UTC
Sent	22 Nov 2012, 5:14:12 UTC
Report deadline	21 Feb 2013, 12:41:23 UTC
Received	15 Dec 2012, 3:52:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1140527
Run time	13 days 3 hours 19 min 15 sec
CPU time	10 days 14 hours 12 min 44 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:22:54 (4494): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:53 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:08 (7005): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:01 (7496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:02 (7496): No heartbeat from core client for 30 sec - exiting 03:46:03 (7496): No heartbeat from core client for 30 sec - exiting 03:46:04 (7496): No heartbeat from core client for 30 sec - exiting 03:46:05 (7496): No heartbeat from core client for 30 sec - exiting 03:46:06 (7496): No heartbeat from core client for 30 sec - exiting 03:46:07 (7496): No heartbeat from core client for 30 sec - exiting 04:11:58 (7623): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:59 (7623): No heartbeat from core client for 30 sec - exiting 04:12:00 (7623): No heartbeat from core client for 30 sec - exiting 04:12:01 (7623): No heartbeat from core client for 30 sec - exiting 04:12:02 (7623): No heartbeat from core client for 30 sec - exiting 04:12:03 (7623): No heartbeat from core client for 30 sec - exiting 04:12:04 (7623): No heartbeat from core client for 30 sec - exiting 04:12:05 (7623): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf775b400] [0xf775b430] /lib32/libc.so.6(gsignal+0x51)[0xf75c3951] /lib32/libc.so.6(abort+0x182)[0xf75c6d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75afbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf777b400] [0xf777b430] /lib32/libc.so.6(gsignal+0x51)[0xf75e3951] /lib32/libc.so.6(abort+0x182)[0xf75e6d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75cfbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7798400] [0xf7798430] /lib32/libc.so.6(gsignal+0x51)[0xf7600951] /lib32/libc.so.6(abort+0x182)[0xf7603d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75ecbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf771b400] [0xf771b430] /lib32/libc.so.6(gsignal+0x51)[0xf7583951] /lib32/libc.so.6(abort+0x182)[0xf7586d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf756fbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76ee400] [0xf76ee430] /lib32/libc.so.6(gsignal+0x51)[0xf7556951] /lib32/libc.so.6(abort+0x182)[0xf7559d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7542bd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf779d400] [0xf779d430] /lib32/libc.so.6(gsignal+0x51)[0xf7605951] /lib32/libc.so.6(abort+0x182)[0xf7608d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75f1bd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9342, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2012 03:52:37	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	492,480	915,042	1.8580
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	466,560	866,990	1.8583
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	440,640	820,942	1.8631
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	414,720	773,580	1.8653
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	388,800	727,678	1.8716
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	362,880	682,341	1.8803
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	336,960	635,226	1.8852
14 Dec 2012 14:47:39	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	311,040	587,628	1.8892
08 Dec 2012 03:25:12	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	285,120	539,007	1.8905
07 Dec 2012 13:52:51	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	259,200	492,084	1.8985
06 Dec 2012 18:21:34	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	233,280	445,475	1.9096
05 Dec 2012 06:44:02	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	207,360	396,178	1.9106
04 Dec 2012 01:20:33	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	181,440	346,760	1.9112
02 Dec 2012 13:54:35	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	155,520	296,427	1.9060
01 Dec 2012 05:50:36	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	129,600	246,098	1.8989
30 Nov 2012 00:24:53	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	103,680	195,377	1.8844
29 Nov 2012 03:16:19	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	77,760	149,829	1.9268
26 Nov 2012 08:45:08	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	51,840	101,298	1.9541
23 Nov 2012 05:34:51	1140527	15451589	hadcm3n_zeqp_1880_40_008250380_1	25,920	51,821	1.9993