Task 13128920

Name	hadcm3n_ymw1_1900_40_007362523_1
Workunit	7559953
Created	6 Jul 2011, 15:26:13 UTC
Sent	6 Jul 2011, 18:20:47 UTC
Report deadline	6 Oct 2011, 1:47:58 UTC
Received	28 Jul 2011, 19:24:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	931768
Run time	6 days 9 hours 10 min 11 sec
CPU time	6 days 9 hours 10 min 11 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.19</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:40:43 (704): No heartbeat from core client for 30 sec - exiting 00:40:44 (704): No heartbeat from core client for 30 sec - exiting 00:40:45 (704): No heartbeat from core client for 30 sec - exiting 00:40:46 (704): No heartbeat from core client for 30 sec - exiting 00:40:47 (704): No heartbeat from core client for 30 sec - exiting 00:40:48 (704): No heartbeat from core client for 30 sec - exiting 00:40:49 (704): No heartbeat from core client for 30 sec - exiting 00:40:50 (704): No heartbeat from core client for 30 sec - exiting 00:40:52 (704): No heartbeat from core client for 30 sec - exiting 00:40:53 (704): No heartbeat from core client for 30 sec - exiting 00:40:54 (704): No heartbeat from core client for 30 sec - exiting 00:40:55 (704): No heartbeat from core client for 30 sec - exiting 00:40:56 (704): No heartbeat from core client for 30 sec - exiting 00:40:57 (704): No heartbeat from core client for 30 sec - exiting 00:40:58 (704): No heartbeat from core client for 30 sec - exiting 00:40:59 (704): No heartbeat from core client for 30 sec - exiting 00:41:00 (704): No heartbeat from core client for 30 sec - exiting 00:41:01 (704): No heartbeat from core client for 30 sec - exiting 00:41:03 (704): No heartbeat from core client for 30 sec - exiting 00:41:04 (704): No heartbeat from core client for 30 sec - exiting 00:41:05 (704): No heartbeat from core client for 30 sec - exiting 00:41:06 (704): No heartbeat from core client for 30 sec - exiting 00:41:07 (704): No heartbeat from core client for 30 sec - exiting 00:41:08 (704): No heartbeat from core client for 30 sec - exiting 00:41:09 (704): No heartbeat from core client for 30 sec - exiting 00:41:10 (704): No heartbeat from core client for 30 sec - exiting 00:41:11 (704): No heartbeat from core client for 30 sec - exiting 00:41:12 (704): No heartbeat from core client for 30 sec - exiting 00:41:13 (704): No heartbeat from core client for 30 sec - exiting 00:41:15 (704): No heartbeat from core client for 30 sec - exiting 00:41:16 (704): No heartbeat from core client for 30 sec - exiting 00:41:17 (704): No heartbeat from core client for 30 sec - exiting 00:41:18 (704): No heartbeat from core client for 30 sec - exiting 00:41:19 (704): No heartbeat from core client for 30 sec - exiting 00:41:20 (704): No heartbeat from core client for 30 sec - exiting 00:41:21 (704): No heartbeat from core client for 30 sec - exiting 00:41:22 (704): No heartbeat from core client for 30 sec - exiting 00:41:23 (704): No heartbeat from core client for 30 sec - exiting 00:41:24 (704): No heartbeat from core client for 30 sec - exiting 00:41:25 (704): No heartbeat from core client for 30 sec - exiting 00:41:27 (704): No heartbeat from core client for 30 sec - exiting 00:41:28 (704): No heartbeat from core client for 30 sec - exiting 00:41:29 (704): No heartbeat from core client for 30 sec - exiting 00:41:30 (704): No heartbeat from core client for 30 sec - exiting 00:41:31 (704): No heartbeat from core client for 30 sec - exiting 00:41:32 (704): No heartbeat from core client for 30 sec - exiting 00:41:33 (704): No heartbeat from core client for 30 sec - exiting 00:41:34 (704): No heartbeat from core client for 30 sec - exiting 00:41:35 (704): No heartbeat from core client for 30 sec - exiting 00:41:36 (704): No heartbeat from core client for 30 sec - exiting 00:41:38 (704): No heartbeat from core client for 30 sec - exiting 00:41:39 (704): No heartbeat from core client for 30 sec - exiting 00:41:40 (704): No heartbeat from core client for 30 sec - exiting 00:41:41 (704): No heartbeat from core client for 30 sec - exiting 00:41:42 (704): No heartbeat from core client for 30 sec - exiting 00:41:43 (704): No heartbeat from core client for 30 sec - exiting 00:41:44 (704): No heartbeat from core client for 30 sec - exiting 00:41:45 (704): No heartbeat from core client for 30 sec - exiting 00:41:46 (704): No heartbeat from core client for 30 sec - exiting 00:41:47 (704): No heartbeat from core client for 30 sec - exiting 00:41:48 (704): No heartbeat from core client for 30 sec - exiting 00:41:50 (704): No heartbeat from core client for 30 sec - exiting 00:41:51 (704): No heartbeat from core client for 30 sec - exiting 00:41:52 (704): No heartbeat from core client for 30 sec - exiting 00:41:53 (704): No heartbeat from core client for 30 sec - exiting 00:41:54 (704): No heartbeat from core client for 30 sec - exiting 00:41:55 (704): No heartbeat from core client for 30 sec - exiting 00:41:56 (704): No heartbeat from core client for 30 sec - exiting 00:41:57 (704): No heartbeat from core client for 30 sec - exiting 00:41:58 (704): No heartbeat from core client for 30 sec - exiting 00:41:59 (704): No heartbeat from core client for 30 sec - exiting 00:42:00 (704): No heartbeat from core client for 30 sec - exiting 00:42:02 (704): No heartbeat from core client for 30 sec - exiting 00:42:03 (704): No heartbeat from core client for 30 sec - exiting 00:42:04 (704): No heartbeat from core client for 30 sec - exiting 00:42:05 (704): No heartbeat from core client for 30 sec - exiting 00:42:06 (704): No heartbeat from core client for 30 sec - exiting 00:42:07 (704): No heartbeat from core client for 30 sec - exiting 00:42:08 (704): No heartbeat from core client for 30 sec - exiting 00:42:09 (704): No heartbeat from core client for 30 sec - exiting 00:42:10 (704): No heartbeat from core client for 30 sec - exiting 00:42:11 (704): No heartbeat from core client for 30 sec - exiting 00:42:13 (704): No heartbeat from core client for 30 sec - exiting 00:42:14 (704): No heartbeat from core client for 30 sec - exiting 00:42:15 (704): No heartbeat from core client for 30 sec - exiting 00:42:16 (704): No heartbeat from core client for 30 sec - exiting 00:42:17 (704): No heartbeat from core client for 30 sec - exiting 00:42:18 (704): No heartbeat from core client for 30 sec - exiting 00:42:19 (704): No heartbeat from core client for 30 sec - exiting 00:42:20 (704): No heartbeat from core client for 30 sec - exiting 00:42:21 (704): No heartbeat from core client for 30 sec - exiting 00:42:22 (704): No heartbeat from core client for 30 sec - exiting 00:42:23 (704): No heartbeat from core client for 30 sec - exiting 00:42:25 (704): No heartbeat from core client for 30 sec - exiting 00:42:26 (704): No heartbeat from core client for 30 sec - exiting 00:42:27 (704): No heartbeat from core client for 30 sec - exiting 00:42:28 (704): No heartbeat from core client for 30 sec - exiting 00:42:29 (704): No heartbeat from core client for 30 sec - exiting 00:42:30 (704): No heartbeat from core client for 30 sec - exiting 00:42:31 (704): No heartbeat from core client for 30 sec - exiting 00:42:32 (704): No heartbeat from core client for 30 sec - exiting 00:42:33 (704): No heartbeat from core client for 30 sec - exiting 00:42:34 (704): No heartbeat from core client for 30 sec - exiting 00:42:35 (704): No heartbeat from core client for 30 sec - exiting 00:42:37 (704): No heartbeat from core client for 30 sec - exiting 00:42:38 (704): No heartbeat from core client for 30 sec - exiting 00:42:39 (704): No heartbeat from core client for 30 sec - exiting 00:42:40 (704): No heartbeat from core client for 30 sec - exiting 00:42:41 (704): No heartbeat from core client for 30 sec - exiting 00:42:42 (704): No heartbeat from core client for 30 sec - exiting 00:42:43 (704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:44 (704): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ymw1ko.pja4c10 Error converting file to netcdf: dataout/ymw1ko.pia4c10 Error converting file to netcdf: dataout/ymw1ko.pfa4c10 Error converting file to netcdf: dataout/ymw1ka.pha4c10 Error converting file to netcdf: dataout/ymw1ka.pga4c10 Error converting file to netcdf: dataout/ymw1ka.pea4c10 Error converting file to netcdf: dataout/ymw1ka.pda4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:03:57	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	207,360	516,558	2.4911
25 Jul 2011 17:13:38	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	181,440	450,509	2.4830
25 Jul 2011 15:50:12	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	155,520	383,737	2.4674
25 Jul 2011 12:58:45	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	129,600	319,567	2.4658
25 Jul 2011 12:58:45	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	103,680	253,810	2.4480
25 Jul 2011 12:58:45	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	77,760	190,698	2.4524
09 Jul 2011 18:40:36	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	51,840	127,645	2.4623
08 Jul 2011 05:42:32	931768	13128920	hadcm3n_ymw1_1900_40_007362523_1	25,920	63,119	2.4351