Task 17784576

Name	hadam3pm2_k12l_1959_10_009444816_2
Workunit	9527050
Created	13 Jan 2015, 10:50:39 UTC
Sent	13 Jan 2015, 14:48:45 UTC
Report deadline	26 Dec 2015, 20:08:45 UTC
Received	6 Feb 2015, 18:59:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1335365
Run time	6 days 7 hours 41 min 43 sec
CPU time	3 days 16 hours 14 min 2 sec
Validate state	Invalid
Credit	6,072.64
Device peak FLOPS	3.29 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Signal 1 received, exiting... 17:47:26 (26990): called boinc_finish 17:47:26 (26990): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|1dec.nc oa.pc\|2jan.nc oa.pc\|2feb.nc oa.pc\|2mar.nc oa.pc\|2apr.nc oa.pc\|2may.nc oa.pc\|2jun.nc oa.pc\|2jul.nc oa.pc\|2aug.nc oa.pc\|2sep.nc oa.pc\|2oct.nc oa.pc\|2nov.nc oa.pe\|1dec.nc oa.pe\|2jan.nc oa.pe\|2feb.nc oa.pe\|2mar.nc oa.pe\|2apr.nc oa.pe\|2may.nc oa.pe\|2jun.nc oa.pe\|2jul.nc oa.pe\|2aug.nc oa.pe\|2sep.nc oa.pe\|2oct.nc oa.pe\|2nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... 11:21:36 (7619): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0efc400] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0cf99d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19220, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f5a400] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0d579d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19220, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... Signal 2 received, exiting... 09:51:29 (19220): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0fd3400] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b06] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0dd09d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19192, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0ef2400] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /data/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0cef9d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19192, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|4dec.nc oa.pc\|5jan.nc oa.pc\|5feb.nc oa.pc\|5mar.nc oa.pc\|5apr.nc oa.pc\|5may.nc oa.pc\|5jun.nc oa.pc\|5jul.nc oa.pc\|5aug.nc oa.pc\|5sep.nc oa.pc\|5oct.nc oa.pc\|5nov.nc oa.pe\|4dec.nc oa.pe\|5jan.nc oa.pe\|5feb.nc oa.pe\|5mar.nc oa.pe\|5apr.nc oa.pe\|5may.nc oa.pe\|5jun.nc oa.pe\|5jul.nc oa.pe\|5aug.nc oa.pe\|5sep.nc oa.pe\|5oct.nc oa.pe\|5nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... 21:58:57 (8475): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... 21:54:36 (24506): called boinc_finish Signal 2 received, exiting... 21:54:37 (24494): called boinc_finish Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Sorry, too many model crashes! :-( 09:57:50 (9102): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Feb 2015 16:57:03	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	210,308	294,477	1.4002
04 Feb 2015 08:50:14	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	204,548	283,519	1.3861
04 Feb 2015 05:04:03	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	201,668	277,964	1.3783
04 Feb 2015 02:06:12	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	198,788	272,280	1.3697
03 Feb 2015 22:13:19	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	195,908	266,536	1.3605
03 Feb 2015 18:07:29	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	193,028	260,975	1.3520
03 Feb 2015 15:16:53	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	190,148	255,391	1.3431
02 Feb 2015 20:04:46	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	187,268	342,923	1.8312
02 Feb 2015 16:28:47	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	184,388	337,529	1.8305
02 Feb 2015 13:37:55	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	181,508	332,012	1.8292
02 Feb 2015 09:49:51	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	178,628	326,612	1.8284
02 Feb 2015 05:43:51	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	175,748	321,241	1.8279
01 Feb 2015 23:10:36	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	169,988	310,219	1.8249
01 Feb 2015 19:20:01	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	167,108	304,726	1.8235
01 Feb 2015 16:54:36	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	164,228	299,277	1.8223
01 Feb 2015 13:03:52	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	161,348	293,714	1.8204
01 Feb 2015 09:13:13	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	158,468	288,132	1.8182
01 Feb 2015 06:27:43	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	155,588	282,614	1.8164
01 Feb 2015 02:51:06	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	152,708	277,200	1.8152
01 Feb 2015 00:04:29	1335365	17784576	hadam3pm2_k12l_1959_10_009444816_2	149,828	271,701	1.8134