Task 13375657

Name	hadcm3n_p1mu_1940_40_007419812_2
Workunit	7617447
Created	11 Sep 2011, 20:34:07 UTC
Sent	11 Sep 2011, 20:40:03 UTC
Report deadline	12 Dec 2011, 4:07:14 UTC
Received	27 Dec 2011, 18:22:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	972749
Run time	46 days 6 hours 23 min 22 sec
CPU time	36 days 0 hours 33 min 34 sec
Validate state	Invalid
Credit	8,709.12
Device peak FLOPS	1.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x1f9400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fc382] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f0c06] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8142ae4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f4942] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807e843] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x8fbbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1331, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xfb3400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fc382] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f0c06] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8142ae4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f4942] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807e843] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xa77bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1288, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (17 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xfd4400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8220265] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8222717] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83104c9] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xa01bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1288, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (12 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xbae400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82e12c4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838e4ba] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82dc8ea] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821a0c8] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2578, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (17 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x638400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8220265] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8222717] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83104c9] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x69fbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1292, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb24400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fc382] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f0c06] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8142ae4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f4942] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807e843] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x147bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1322, iMonCtr=1 Model crash detected, will try to restart... Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Model crashed: TRANSO2A: Missing data in ocean UV fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 02:03:53	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	725,760	3,225,335	4.4441
24 Dec 2011 10:05:09	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	699,840	3,112,465	4.4474
22 Dec 2011 17:51:24	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	673,920	2,969,915	4.4069
18 Nov 2011 17:41:38	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	648,000	2,828,795	4.3654
17 Nov 2011 01:51:36	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	622,080	2,686,744	4.3190
15 Nov 2011 17:31:01	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	596,160	2,543,023	4.2657
08 Nov 2011 03:32:04	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	570,240	2,400,568	4.2098
06 Nov 2011 11:47:54	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	544,320	2,446,578	4.4947
04 Nov 2011 20:37:10	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	518,400	2,302,558	4.4417
31 Oct 2011 19:16:22	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	492,480	2,167,245	4.4007
31 Oct 2011 18:22:54	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	466,560	2,033,972	4.3595
31 Oct 2011 17:00:59	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	440,640	1,951,113	4.4279
31 Oct 2011 16:38:23	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	414,720	1,877,990	4.5283
31 Oct 2011 16:38:23	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	388,800	1,734,864	4.4621
16 Oct 2011 17:49:29	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	362,880	1,591,555	4.3859
15 Oct 2011 01:41:13	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	336,960	1,448,375	4.2984
11 Oct 2011 11:27:18	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	311,040	1,694,536	5.4480
09 Oct 2011 19:35:37	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	285,120	1,552,034	5.4434
08 Oct 2011 03:37:22	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	259,200	1,409,609	5.4383
05 Oct 2011 10:06:20	972749	13375657	hadcm3n_p1mu_1940_40_007419812_2	233,280	1,266,597	5.4295