Task 18931460

Name	hadam3pm2_a2in_1959_10_008649450_1
Workunit	8795924
Created	19 Sep 2015, 12:22:43 UTC
Sent	19 Sep 2015, 20:26:07 UTC
Report deadline	1 Sep 2016, 1:46:07 UTC
Received	9 Oct 2015, 20:50:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	517888
Run time	12 days 7 hours 28 min 21 sec
CPU time	9 days 15 hours 13 min 51 sec
Validate state	Invalid
Credit	8,401.12
Device peak FLOPS	3.93 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc SIGSEGV: segmentation violation Stack trace (13 frames): /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0ee6e50] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0ce9993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=911, iMonCtr=1 Model crash detected, will try to restart... oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f3be50] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b06] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0d3e993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6006, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 1 received, exiting... Signal 1 received, exiting... 22:47:01 (3298): called boinc_finish SIGSEGV: segmentation violation Stack trace (13 frames): /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f17e50] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b06] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0d1a993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19374, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0efbe50] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/jordi/fonts/seti/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0cfe993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Sorry, too many model crashes! :-( 22:11:23 (2295): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2015 19:52:15	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	290,948	832,361	2.8609
09 Oct 2015 18:16:51	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	288,068	826,531	2.8692
09 Oct 2015 17:06:36	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	285,188	820,574	2.8773
09 Oct 2015 15:01:06	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	282,308	815,156	2.8875
09 Oct 2015 13:20:39	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	279,428	808,834	2.8946
09 Oct 2015 10:39:42	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	273,668	797,131	2.9128
09 Oct 2015 08:28:01	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	270,788	791,650	2.9235
09 Oct 2015 07:12:06	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	267,908	786,237	2.9347
09 Oct 2015 06:11:23	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	265,028	780,674	2.9456
09 Oct 2015 04:14:46	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	262,148	774,646	2.9550
09 Oct 2015 03:13:45	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	259,268	769,444	2.9678
09 Oct 2015 01:17:50	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	256,388	764,140	2.9804
09 Oct 2015 00:10:46	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	253,508	759,693	2.9967
08 Oct 2015 23:10:31	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	250,628	754,859	3.0119
08 Oct 2015 21:05:03	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	247,748	749,799	3.0265
08 Oct 2015 20:04:49	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	244,868	744,902	3.0421
08 Oct 2015 16:48:57	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	239,108	732,839	3.0649
08 Oct 2015 14:48:27	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	236,228	725,969	3.0732
08 Oct 2015 12:47:49	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	233,348	719,002	3.0812
08 Oct 2015 10:21:43	517888	18931460	hadam3pm2_a2in_1959_10_008649450_1	230,468	712,437	3.0913