Task 16573276

Name	hadam3pm2_e8xs_1991_10_008714627_0
Workunit	8861101
Created	19 Apr 2014, 9:28:42 UTC
Sent	22 Apr 2014, 17:01:13 UTC
Report deadline	4 Apr 2015, 22:21:13 UTC
Received	12 May 2014, 8:13:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1258512
Run time	19 days 8 hours 33 min 3 sec
CPU time	15 days 2 hours 27 min 23 sec
Validate state	Invalid
Credit	8,900.08
Device peak FLOPS	2.37 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> oa.pc\|1dec.nc oa.pc\|2jan.nc oa.pc\|2feb.nc oa.pc\|2mar.nc oa.pc\|2apr.nc oa.pc\|2may.nc oa.pc\|2jun.nc oa.pc\|2jul.nc oa.pc\|2aug.nc oa.pc\|2sep.nc oa.pc\|2oct.nc oa.pc\|2nov.nc oa.pe\|1dec.nc oa.pe\|2jan.nc oa.pe\|2feb.nc oa.pe\|2mar.nc oa.pe\|2apr.nc oa.pe\|2may.nc oa.pe\|2jun.nc oa.pe\|2jul.nc oa.pe\|2aug.nc oa.pe\|2sep.nc oa.pe\|2oct.nc oa.pe\|2nov.nc oa.pc\|2dec.nc oa.pc\|3jan.nc oa.pc\|3feb.nc oa.pc\|3mar.nc oa.pc\|3apr.nc oa.pc\|3may.nc oa.pc\|3jun.nc oa.pc\|3jul.nc oa.pc\|3aug.nc oa.pc\|3sep.nc oa.pc\|3oct.nc oa.pc\|3nov.nc oa.pe\|2dec.nc oa.pe\|3jan.nc oa.pe\|3feb.nc oa.pe\|3mar.nc oa.pe\|3apr.nc oa.pe\|3may.nc oa.pe\|3jun.nc oa.pe\|3jul.nc oa.pe\|3aug.nc oa.pe\|3sep.nc oa.pe\|3oct.nc oa.pe\|3nov.nc oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc oa.pc\|4dec.nc oa.pc\|5jan.nc oa.pc\|5feb.nc oa.pc\|5mar.nc oa.pc\|5apr.nc oa.pc\|5may.nc oa.pc\|5jun.nc oa.pc\|5jul.nc oa.pc\|5aug.nc oa.pc\|5sep.nc oa.pc\|5oct.nc oa.pc\|5nov.nc oa.pe\|4dec.nc oa.pe\|5jan.nc oa.pe\|5feb.nc oa.pe\|5mar.nc oa.pe\|5apr.nc oa.pe\|5may.nc oa.pe\|5jun.nc oa.pe\|5jul.nc oa.pe\|5aug.nc oa.pe\|5sep.nc oa.pe\|5oct.nc oa.pe\|5nov.nc oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc 03:07:31 (6931): No heartbeat from client for 30 sec - exiting 03:07:31 (6931): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:10 (11488): No heartbeat from client for 30 sec - exiting 03:08:10 (11488): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:32 (11512): No heartbeat from client for 30 sec - exiting 03:10:32 (11512): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:13:39 (11552): No heartbeat from client for 30 sec - exiting 03:13:39 (11552): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:14:23 (11577): No heartbeat from client for 30 sec - exiting 03:14:23 (11577): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:40 (11601): No heartbeat from client for 30 sec - exiting 03:16:40 (11601): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:17:24 (11626): No heartbeat from client for 30 sec - exiting 03:17:24 (11626): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:42 (11655): No heartbeat from client for 30 sec - exiting 03:19:42 (11655): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:26 (11689): No heartbeat from client for 30 sec - exiting 03:20:26 (11689): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:27 (11714): No heartbeat from client for 30 sec - exiting 03:23:27 (11714): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:13 (11737): No heartbeat from client for 30 sec - exiting 03:24:13 (11737): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:29 (11759): No heartbeat from client for 30 sec - exiting 03:26:29 (11759): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:14 (11781): No heartbeat from client for 30 sec - exiting 03:27:14 (11781): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:29:29 (11803): No heartbeat from client for 30 sec - exiting 03:29:29 (11803): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:16 (11835): No heartbeat from client for 30 sec - exiting 03:30:16 (11835): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:02 (11858): No heartbeat from client for 30 sec - exiting 03:31:02 (11858): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:33:17 (11880): No heartbeat from client for 30 sec - exiting 03:33:17 (11880): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:03 (11903): No heartbeat from client for 30 sec - exiting 03:34:03 (11903): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:05 (11925): No heartbeat from client for 30 sec - exiting 03:37:05 (11925): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:50 (11948): No heartbeat from client for 30 sec - exiting 03:37:50 (11948): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:06 (11970): No heartbeat from client for 30 sec - exiting 03:40:06 (11970): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:52 (12002): No heartbeat from client for 30 sec - exiting 03:40:52 (12002): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:07 (12025): No heartbeat from client for 30 sec - exiting 03:43:07 (12025): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:53 (12048): No heartbeat from client for 30 sec - exiting 03:43:53 (12048): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:39 (12081): No heartbeat from client for 30 sec - exiting 03:44:39 (12081): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:54 (12103): No heartbeat from client for 30 sec - exiting 03:46:54 (12103): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:40 (12127): No heartbeat from client for 30 sec - exiting 03:47:40 (12127): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... oa.pc\|8dec.nc oa.pc\|9jan.nc oa.pc\|9feb.nc oa.pc\|9mar.nc oa.pc\|9apr.nc oa.pc\|9may.nc oa.pc\|9jun.nc oa.pc\|9jul.nc oa.pc\|9aug.nc oa.pc\|9sep.nc oa.pc\|9oct.nc oa.pc\|9nov.nc oa.pe\|8dec.nc oa.pe\|9jan.nc oa.pe\|9feb.nc oa.pe\|9mar.nc oa.pe\|9apr.nc oa.pe\|9may.nc oa.pe\|9jun.nc oa.pe\|9jul.nc oa.pe\|9aug.nc oa.pe\|9sep.nc oa.pe\|9oct.nc oa.pe\|9nov.nc oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Sorry, too many model crashes! :-( 05:00:09 (12149): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 May 2014 19:12:45	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	308,228	1,626,016	5.2754
11 May 2014 15:29:48	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	305,348	1,612,765	5.2817
11 May 2014 11:58:45	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	302,468	1,599,954	5.2897
11 May 2014 08:22:49	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	299,588	1,587,283	5.2982
11 May 2014 04:45:35	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	296,708	1,574,265	5.3058
11 May 2014 01:09:40	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	293,828	1,561,161	5.3132
10 May 2014 21:38:45	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	290,948	1,548,487	5.3222
10 May 2014 18:02:43	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	288,068	1,535,618	5.3307
10 May 2014 14:29:24	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	285,188	1,522,764	5.3395
10 May 2014 10:55:32	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	282,308	1,509,982	5.3487
10 May 2014 07:23:39	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	279,428	1,497,360	5.3587
10 May 2014 00:16:29	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	273,668	1,471,785	5.3780
09 May 2014 20:35:18	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	270,788	1,458,675	5.3868
09 May 2014 16:58:57	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	267,908	1,445,744	5.3964
09 May 2014 13:20:20	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	265,028	1,432,554	5.4053
09 May 2014 09:38:29	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	262,148	1,419,269	5.4140
09 May 2014 05:55:18	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	259,268	1,406,007	5.4230
09 May 2014 02:14:27	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	256,388	1,392,701	5.4320
08 May 2014 22:28:27	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	253,508	1,379,272	5.4407
08 May 2014 18:42:11	1258512	16573276	hadam3pm2_e8xs_1991_10_008714627_0	250,628	1,365,593	5.4487