Task 18960386

Name	hadam3pm2_c53x_1975_10_008676808_1
Workunit	8832138
Created	20 Sep 2015, 13:08:40 UTC
Sent	21 Sep 2015, 23:36:27 UTC
Report deadline	3 Sep 2016, 4:56:27 UTC
Received	11 Oct 2015, 0:26:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1292459
Run time	5 days 1 hours 23 min 30 sec
CPU time	6 min 7 sec
Validate state	Invalid
Credit	5,906.32
Device peak FLOPS	3.55 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|7dec.nc oa.pc\|8jan.nc oa.pc\|8feb.nc oa.pc\|8mar.nc oa.pc\|8apr.nc oa.pc\|8may.nc oa.pc\|8jun.nc oa.pc\|8jul.nc oa.pc\|8aug.nc oa.pc\|8sep.nc oa.pc\|8oct.nc oa.pc\|8nov.nc oa.pe\|7dec.nc oa.pe\|8jan.nc oa.pe\|8feb.nc oa.pe\|8mar.nc oa.pe\|8apr.nc oa.pe\|8may.nc oa.pe\|8jun.nc oa.pe\|8jul.nc oa.pe\|8aug.nc oa.pe\|8sep.nc oa.pe\|8oct.nc oa.pe\|8nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|8dec.nc oa.pc\|9jan.nc oa.pc\|9feb.nc oa.pc\|9mar.nc oa.pc\|9apr.nc oa.pc\|9may.nc oa.pc\|9jun.nc oa.pc\|9jul.nc oa.pc\|9aug.nc oa.pc\|9sep.nc oa.pc\|9oct.nc oa.pc\|9nov.nc oa.pe\|8dec.nc oa.pe\|9jan.nc oa.pe\|9feb.nc oa.pe\|9mar.nc oa.pe\|9apr.nc oa.pe\|9may.nc oa.pe\|9jun.nc oa.pe\|9jul.nc oa.pe\|9aug.nc oa.pe\|9sep.nc oa.pe\|9oct.nc oa.pe\|9nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f36be0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0d13efe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f9cbe0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0d79efe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f90be0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0d6defe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f54be0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0d31efe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0fc4be0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0da1efe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f0cbe0] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0xf0ce9efe] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10435, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 02:13:02 (10435): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2015 21:51:06	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	204,548	390,019	1.9067
10 Oct 2015 20:50:55	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	201,668	384,553	1.9069
10 Oct 2015 18:50:31	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	198,788	379,144	1.9073
10 Oct 2015 16:50:06	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	195,908	373,703	1.9075
10 Oct 2015 15:49:54	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	193,028	368,262	1.9078
10 Oct 2015 13:44:25	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	190,148	362,828	1.9081
10 Oct 2015 11:43:55	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	187,268	357,231	1.9076
10 Oct 2015 09:43:16	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	184,388	351,586	1.9068
10 Oct 2015 06:41:35	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	181,508	346,105	1.9068
10 Oct 2015 04:40:29	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	178,628	340,579	1.9066
10 Oct 2015 02:38:30	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	175,748	335,173	1.9071
09 Oct 2015 21:37:37	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	169,988	324,085	1.9065
09 Oct 2015 19:32:09	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	167,108	318,616	1.9066
09 Oct 2015 17:31:42	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	164,228	312,747	1.9043
09 Oct 2015 14:30:59	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	161,348	307,113	1.9034
09 Oct 2015 12:30:24	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	158,468	301,609	1.9033
09 Oct 2015 11:30:06	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	155,588	296,127	1.9033
09 Oct 2015 09:28:59	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	152,708	290,650	1.9033
09 Oct 2015 07:27:15	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	149,828	285,214	1.9036
09 Oct 2015 05:15:36	1292459	18960386	hadam3pm2_c53x_1975_10_008676808_1	146,948	279,700	1.9034