Task 13096765

Name	hadcm3n_yahj_1900_40_007346449_1
Workunit	7543879
Created	6 Jul 2011, 13:37:49 UTC
Sent	19 Jul 2011, 17:52:37 UTC
Report deadline	19 Oct 2011, 1:19:48 UTC
Received	11 Sep 2011, 20:49:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1101198
Run time	17 days 20 hours 41 min 52 sec
CPU time	17 days 20 hours 40 min 24 sec
Validate state	Invalid
Credit	9,020.16
Device peak FLOPS	2.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:15 (246): No heartbeat from core client for 30 sec - exiting 14:30:16 (246): No heartbeat from core client for 30 sec - exiting 14:30:17 (246): No heartbeat from core client for 30 sec - exiting 14:30:18 (246): No heartbeat from core client for 30 sec - exiting 14:30:19 (246): No heartbeat from core client for 30 sec - exiting 14:30:20 (246): No heartbeat from core client for 30 sec - exiting 14:30:21 (246): No heartbeat from core client for 30 sec - exiting 14:30:22 (246): No heartbeat from core client for 30 sec - exiting 14:30:23 (246): No heartbeat from core client for 30 sec - exiting 14:30:24 (246): No heartbeat from core client for 30 sec - exiting 14:30:25 (246): No heartbeat from core client for 30 sec - exiting 14:30:26 (246): No heartbeat from core client for 30 sec - exiting 14:30:27 (246): No heartbeat from core client for 30 sec - exiting 14:30:28 (246): No heartbeat from core client for 30 sec - exiting 14:30:29 (246): No heartbeat from core client for 30 sec - exiting 14:30:30 (246): No heartbeat from core client for 30 sec - exiting 14:30:31 (246): No heartbeat from core client for 30 sec - exiting 14:30:32 (246): No heartbeat from core client for 30 sec - exiting 14:30:33 (246): No heartbeat from core client for 30 sec - exiting 14:30:34 (246): No heartbeat from core client for 30 sec - exiting 14:30:35 (246): No heartbeat from core client for 30 sec - exiting 14:30:36 (246): No heartbeat from core client for 30 sec - exiting 14:30:37 (246): No heartbeat from core client for 30 sec - exiting 14:30:38 (246): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:06:16 (150): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:10:09 (25069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:10:10 (25069): No heartbeat from core client for 30 sec - exiting 19:10:11 (25069): No heartbeat from core client for 30 sec - exiting 19:10:12 (25069): No heartbeat from core client for 30 sec - exiting 19:10:13 (25069): No heartbeat from core client for 30 sec - exiting 19:10:14 (25069): No heartbeat from core client for 30 sec - exiting 19:10:15 (25069): No heartbeat from core client for 30 sec - exiting 19:10:16 (25069): No heartbeat from core client for 30 sec - exiting 19:10:17 (25069): No heartbeat from core client for 30 sec - exiting 19:10:18 (25069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:24:14 (144): No heartbeat from core client for 30 sec - exiting 20:24:15 (144): No heartbeat from core client for 30 sec - exiting 20:24:16 (144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:15:39 (181): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:40 (181): No heartbeat from core client for 30 sec - exiting 20:15:41 (181): No heartbeat from core client for 30 sec - exiting 20:15:42 (181): No heartbeat from core client for 30 sec - exiting 20:21:23 (202): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:09:27 (145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 139265) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=960, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Sep 2011 10:13:15	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	751,680	1,515,667	2.0164
10 Sep 2011 15:11:41	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	725,760	1,461,734	2.0141
09 Sep 2011 23:36:28	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	699,840	1,410,904	2.0160
09 Sep 2011 08:15:51	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	673,920	1,359,773	2.0177
08 Sep 2011 17:00:53	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	648,000	1,308,954	2.0200
08 Sep 2011 01:48:55	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	622,080	1,258,013	2.0223
07 Sep 2011 08:30:44	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	596,160	1,206,763	2.0242
06 Sep 2011 08:14:52	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	570,240	1,155,226	2.0259
05 Sep 2011 10:13:39	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	544,320	1,103,989	2.0282
04 Sep 2011 11:27:30	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	518,400	1,053,299	2.0318
03 Sep 2011 11:49:36	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	492,480	1,001,304	2.0332
02 Sep 2011 13:16:52	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	466,560	949,223	2.0345
01 Sep 2011 13:46:38	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	440,640	898,184	2.0384
31 Aug 2011 14:07:45	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	414,720	846,514	2.0412
30 Aug 2011 15:21:40	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	388,800	794,409	2.0432
29 Aug 2011 18:20:31	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	362,880	742,009	2.0448
29 Aug 2011 01:47:05	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	336,960	690,263	2.0485
28 Aug 2011 08:41:28	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	311,040	637,763	2.0504
27 Aug 2011 10:00:22	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	285,120	586,407	2.0567
26 Aug 2011 08:22:49	1101198	13096765	hadcm3n_yahj_1900_40_007346449_1	259,200	532,909	2.0560