Task 13095666

Name	hadcm3n_ya2a_1900_40_007345900_1
Workunit	7543330
Created	6 Jul 2011, 13:34:14 UTC
Sent	20 Jul 2011, 1:47:57 UTC
Report deadline	19 Oct 2011, 9:15:08 UTC
Received	9 Aug 2011, 16:10:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	931768
Run time	10 days 0 hours 38 min 17 sec
CPU time	10 days 0 hours 38 min 17 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	1.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.19</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:47:32 (4876): No heartbeat from core client for 30 sec - exiting 22:47:33 (4876): No heartbeat from core client for 30 sec - exiting 22:47:34 (4876): No heartbeat from core client for 30 sec - exiting 22:47:35 (4876): No heartbeat from core client for 30 sec - exiting 22:47:37 (4876): No heartbeat from core client for 30 sec - exiting 22:47:38 (4876): No heartbeat from core client for 30 sec - exiting 22:47:39 (4876): No heartbeat from core client for 30 sec - exiting 22:47:40 (4876): No heartbeat from core client for 30 sec - exiting 22:47:41 (4876): No heartbeat from core client for 30 sec - exiting 22:47:42 (4876): No heartbeat from core client for 30 sec - exiting 22:47:43 (4876): No heartbeat from core client for 30 sec - exiting 22:47:44 (4876): No heartbeat from core client for 30 sec - exiting 22:47:45 (4876): No heartbeat from core client for 30 sec - exiting 22:47:46 (4876): No heartbeat from core client for 30 sec - exiting 22:47:47 (4876): No heartbeat from core client for 30 sec - exiting 22:47:49 (4876): No heartbeat from core client for 30 sec - exiting 22:47:50 (4876): No heartbeat from core client for 30 sec - exiting 22:47:51 (4876): No heartbeat from core client for 30 sec - exiting 22:47:52 (4876): No heartbeat from core client for 30 sec - exiting 22:47:53 (4876): No heartbeat from core client for 30 sec - exiting 22:47:54 (4876): No heartbeat from core client for 30 sec - exiting 22:47:55 (4876): No heartbeat from core client for 30 sec - exiting 22:47:56 (4876): No heartbeat from core client for 30 sec - exiting 22:47:57 (4876): No heartbeat from core client for 30 sec - exiting 22:47:58 (4876): No heartbeat from core client for 30 sec - exiting 22:48:00 (4876): No heartbeat from core client for 30 sec - exiting 22:48:01 (4876): No heartbeat from core client for 30 sec - exiting 22:48:02 (4876): No heartbeat from core client for 30 sec - exiting 22:48:03 (4876): No heartbeat from core client for 30 sec - exiting 22:48:04 (4876): No heartbeat from core client for 30 sec - exiting 22:48:05 (4876): No heartbeat from core client for 30 sec - exiting 22:48:06 (4876): No heartbeat from core client for 30 sec - exiting 22:48:07 (4876): No heartbeat from core client for 30 sec - exiting 22:48:08 (4876): No heartbeat from core client for 30 sec - exiting 22:48:09 (4876): No heartbeat from core client for 30 sec - exiting 22:48:10 (4876): No heartbeat from core client for 30 sec - exiting 22:48:12 (4876): No heartbeat from core client for 30 sec - exiting 22:48:13 (4876): No heartbeat from core client for 30 sec - exiting 22:48:14 (4876): No heartbeat from core client for 30 sec - exiting 22:48:15 (4876): No heartbeat from core client for 30 sec - exiting 22:48:16 (4876): No heartbeat from core client for 30 sec - exiting 22:48:17 (4876): No heartbeat from core client for 30 sec - exiting 22:48:18 (4876): No heartbeat from core client for 30 sec - exiting 22:48:19 (4876): No heartbeat from core client for 30 sec - exiting 22:48:20 (4876): No heartbeat from core client for 30 sec - exiting 22:48:21 (4876): No heartbeat from core client for 30 sec - exiting 22:48:22 (4876): No heartbeat from core client for 30 sec - exiting 22:48:24 (4876): No heartbeat from core client for 30 sec - exiting 22:48:25 (4876): No heartbeat from core client for 30 sec - exiting 22:48:26 (4876): No heartbeat from core client for 30 sec - exiting 22:48:27 (4876): No heartbeat from core client for 30 sec - exiting 22:48:28 (4876): No heartbeat from core client for 30 sec - exiting 22:48:29 (4876): No heartbeat from core client for 30 sec - exiting 22:48:30 (4876): No heartbeat from core client for 30 sec - exiting 22:48:31 (4876): No heartbeat from core client for 30 sec - exiting 22:48:32 (4876): No heartbeat from core client for 30 sec - exiting 22:48:33 (4876): No heartbeat from core client for 30 sec - exiting 22:48:35 (4876): No heartbeat from core client for 30 sec - exiting 22:48:36 (4876): No heartbeat from core client for 30 sec - exiting 22:48:37 (4876): No heartbeat from core client for 30 sec - exiting 22:48:38 (4876): No heartbeat from core client for 30 sec - exiting 22:48:39 (4876): No heartbeat from core client for 30 sec - exiting 22:48:40 (4876): No heartbeat from core client for 30 sec - exiting 22:48:41 (4876): No heartbeat from core client for 30 sec - exiting 22:48:42 (4876): No heartbeat from core client for 30 sec - exiting 22:48:43 (4876): No heartbeat from core client for 30 sec - exiting 22:48:44 (4876): No heartbeat from core client for 30 sec - exiting 22:48:45 (4876): No heartbeat from core client for 30 sec - exiting 22:48:47 (4876): No heartbeat from core client for 30 sec - exiting 22:48:48 (4876): No heartbeat from core client for 30 sec - exiting 22:48:49 (4876): No heartbeat from core client for 30 sec - exiting 22:48:50 (4876): No heartbeat from core client for 30 sec - exiting 22:48:51 (4876): No heartbeat from core client for 30 sec - exiting 22:48:52 (4876): No heartbeat from core client for 30 sec - exiting 22:48:53 (4876): No heartbeat from core client for 30 sec - exiting 22:48:54 (4876): No heartbeat from core client for 30 sec - exiting 22:48:55 (4876): No heartbeat from core client for 30 sec - exiting 22:48:56 (4876): No heartbeat from core client for 30 sec - exiting 22:48:57 (4876): No heartbeat from core client for 30 sec - exiting 22:48:59 (4876): No heartbeat from core client for 30 sec - exiting 22:49:00 (4876): No heartbeat from core client for 30 sec - exiting 22:49:01 (4876): No heartbeat from core client for 30 sec - exiting 22:49:02 (4876): No heartbeat from core client for 30 sec - exiting 22:49:03 (4876): No heartbeat from core client for 30 sec - exiting 22:49:04 (4876): No heartbeat from core client for 30 sec - exiting 22:49:05 (4876): No heartbeat from core client for 30 sec - exiting 22:49:06 (4876): No heartbeat from core client for 30 sec - exiting 22:49:07 (4876): No heartbeat from core client for 30 sec - exiting 22:49:08 (4876): No heartbeat from core client for 30 sec - exiting 22:49:10 (4876): No heartbeat from core client for 30 sec - exiting 22:49:11 (4876): No heartbeat from core client for 30 sec - exiting 22:49:12 (4876): No heartbeat from core client for 30 sec - exiting 22:49:13 (4876): No heartbeat from core client for 30 sec - exiting 22:49:14 (4876): No heartbeat from core client for 30 sec - exiting 22:49:15 (4876): No heartbeat from core client for 30 sec - exiting 22:49:16 (4876): No heartbeat from core client for 30 sec - exiting 22:49:17 (4876): No heartbeat from core client for 30 sec - exiting 22:49:18 (4876): No heartbeat from core client for 30 sec - exiting 22:49:19 (4876): No heartbeat from core client for 30 sec - exiting 22:49:20 (4876): No heartbeat from core client for 30 sec - exiting 22:49:22 (4876): No heartbeat from core client for 30 sec - exiting 22:49:23 (4876): No heartbeat from core client for 30 sec - exiting 22:49:24 (4876): No heartbeat from core client for 30 sec - exiting 22:49:25 (4876): No heartbeat from core client for 30 sec - exiting 22:49:26 (4876): No heartbeat from core client for 30 sec - exiting 22:49:27 (4876): No heartbeat from core client for 30 sec - exiting 22:49:28 (4876): No heartbeat from core client for 30 sec - exiting 22:49:29 (4876): No heartbeat from core client for 30 sec - exiting 22:49:30 (4876): No heartbeat from core client for 30 sec - exiting 22:49:31 (4876): No heartbeat from core client for 30 sec - exiting 22:49:33 (4876): No heartbeat from core client for 30 sec - exiting 22:49:34 (4876): No heartbeat from core client for 30 sec - exiting 22:49:35 (4876): No heartbeat from core client for 30 sec - exiting 22:49:36 (4876): No heartbeat from core client for 30 sec - exiting 22:49:37 (4876): No heartbeat from core client for 30 sec - exiting 22:49:38 (4876): No heartbeat from core client for 30 sec - exiting 22:49:39 (4876): No heartbeat from core client for 30 sec - exiting 22:49:40 (4876): No heartbeat from core client for 30 sec - exiting 22:49:41 (4876): No heartbeat from core client for 30 sec - exiting 22:49:42 (4876): No heartbeat from core client for 30 sec - exiting 22:49:43 (4876): No heartbeat from core client for 30 sec - exiting 22:49:45 (4876): No heartbeat from core client for 30 sec - exiting 22:49:46 (4876): No heartbeat from core client for 30 sec - exiting 22:49:47 (4876): No heartbeat from core client for 30 sec - exiting 22:49:48 (4876): No heartbeat from core client for 30 sec - exiting 22:49:49 (4876): No heartbeat from core client for 30 sec - exiting 22:49:50 (4876): No heartbeat from core client for 30 sec - exiting 22:49:51 (4876): No heartbeat from core client for 30 sec - exiting 22:49:52 (4876): No heartbeat from core client for 30 sec - exiting 22:49:53 (4876): No heartbeat from core client for 30 sec - exiting 22:49:54 (4876): No heartbeat from core client for 30 sec - exiting 22:49:55 (4876): No heartbeat from core client for 30 sec - exiting 22:49:57 (4876): No heartbeat from core client for 30 sec - exiting 22:49:58 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ya2ako.pjb3c10 Error converting file to netcdf: dataout/ya2ako.pib3c10 Error converting file to netcdf: dataout/ya2ako.pfb3c10 Error converting file to netcdf: dataout/ya2aka.phb3c10 Error converting file to netcdf: dataout/ya2aka.pgb3c10 Error converting file to netcdf: dataout/ya2aka.peb3c10 Error converting file to netcdf: dataout/ya2aka.pdb3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10324, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Aug 2011 03:56:11	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	336,960	845,013	2.5078
07 Aug 2011 16:34:09	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	311,040	782,236	2.5149
06 Aug 2011 03:52:55	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	285,120	717,194	2.5154
04 Aug 2011 12:50:32	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	259,200	650,702	2.5104
03 Aug 2011 01:51:56	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	233,280	585,302	2.5090
01 Aug 2011 12:11:52	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	207,360	519,834	2.5069
30 Jul 2011 20:57:01	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	181,440	452,305	2.4929
29 Jul 2011 13:15:45	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	155,520	386,699	2.4865
28 Jul 2011 19:28:08	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	129,600	321,928	2.4840
28 Jul 2011 19:28:08	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	103,680	255,540	2.4647
28 Jul 2011 19:28:08	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	77,760	193,256	2.4853
28 Jul 2011 19:28:08	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	51,840	127,260	2.4549
28 Jul 2011 19:28:08	931768	13095666	hadcm3n_ya2a_1900_40_007345900_1	25,920	63,945	2.4670