Task 13396440

Name	hadcm3n_t37w_1940_40_007446063_2
Workunit	7643566
Created	18 Sep 2011, 18:36:56 UTC
Sent	18 Sep 2011, 18:51:42 UTC
Report deadline	19 Dec 2011, 2:18:53 UTC
Received	27 Sep 2011, 3:29:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	931768
Run time	2 days 12 hours 36 min 51 sec
CPU time	2 days 12 hours 36 min 51 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	1.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.19</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:19:17 (1912): No heartbeat from core client for 30 sec - exiting 00:19:18 (1912): No heartbeat from core client for 30 sec - exiting 00:19:19 (1912): No heartbeat from core client for 30 sec - exiting 00:19:20 (1912): No heartbeat from core client for 30 sec - exiting 00:19:21 (1912): No heartbeat from core client for 30 sec - exiting 00:19:22 (1912): No heartbeat from core client for 30 sec - exiting 00:19:23 (1912): No heartbeat from core client for 30 sec - exiting 00:19:24 (1912): No heartbeat from core client for 30 sec - exiting 00:19:25 (1912): No heartbeat from core client for 30 sec - exiting 00:19:26 (1912): No heartbeat from core client for 30 sec - exiting 00:19:27 (1912): No heartbeat from core client for 30 sec - exiting 00:19:28 (1912): No heartbeat from core client for 30 sec - exiting 00:19:29 (1912): No heartbeat from core client for 30 sec - exiting 00:19:30 (1912): No heartbeat from core client for 30 sec - exiting 00:19:31 (1912): No heartbeat from core client for 30 sec - exiting 00:19:32 (1912): No heartbeat from core client for 30 sec - exiting 00:19:33 (1912): No heartbeat from core client for 30 sec - exiting 00:19:34 (1912): No heartbeat from core client for 30 sec - exiting 00:19:35 (1912): No heartbeat from core client for 30 sec - exiting 00:19:36 (1912): No heartbeat from core client for 30 sec - exiting 00:19:37 (1912): No heartbeat from core client for 30 sec - exiting 00:19:38 (1912): No heartbeat from core client for 30 sec - exiting 00:19:39 (1912): No heartbeat from core client for 30 sec - exiting 00:19:40 (1912): No heartbeat from core client for 30 sec - exiting 00:19:41 (1912): No heartbeat from core client for 30 sec - exiting 00:19:42 (1912): No heartbeat from core client for 30 sec - exiting 00:19:43 (1912): No heartbeat from core client for 30 sec - exiting 00:19:44 (1912): No heartbeat from core client for 30 sec - exiting 00:19:45 (1912): No heartbeat from core client for 30 sec - exiting 00:19:46 (1912): No heartbeat from core client for 30 sec - exiting 00:19:47 (1912): No heartbeat from core client for 30 sec - exiting 00:19:48 (1912): No heartbeat from core client for 30 sec - exiting 00:19:49 (1912): No heartbeat from core client for 30 sec - exiting 00:19:50 (1912): No heartbeat from core client for 30 sec - exiting 00:19:51 (1912): No heartbeat from core client for 30 sec - exiting 00:19:52 (1912): No heartbeat from core client for 30 sec - exiting 00:19:53 (1912): No heartbeat from core client for 30 sec - exiting 00:19:54 (1912): No heartbeat from core client for 30 sec - exiting 00:19:55 (1912): No heartbeat from core client for 30 sec - exiting 00:19:56 (1912): No heartbeat from core client for 30 sec - exiting 00:19:57 (1912): No heartbeat from core client for 30 sec - exiting 00:19:58 (1912): No heartbeat from core client for 30 sec - exiting 00:19:59 (1912): No heartbeat from core client for 30 sec - exiting 00:20:00 (1912): No heartbeat from core client for 30 sec - exiting 00:20:01 (1912): No heartbeat from core client for 30 sec - exiting 00:20:02 (1912): No heartbeat from core client for 30 sec - exiting 00:20:03 (1912): No heartbeat from core client for 30 sec - exiting 00:20:04 (1912): No heartbeat from core client for 30 sec - exiting 00:20:05 (1912): No heartbeat from core client for 30 sec - exiting 00:20:06 (1912): No heartbeat from core client for 30 sec - exiting 00:20:07 (1912): No heartbeat from core client for 30 sec - exiting 00:20:08 (1912): No heartbeat from core client for 30 sec - exiting 00:20:09 (1912): No heartbeat from core client for 30 sec - exiting 00:20:10 (1912): No heartbeat from core client for 30 sec - exiting 00:20:11 (1912): No heartbeat from core client for 30 sec - exiting 00:20:12 (1912): No heartbeat from core client for 30 sec - exiting 00:20:13 (1912): No heartbeat from core client for 30 sec - exiting 00:20:14 (1912): No heartbeat from core client for 30 sec - exiting 00:20:15 (1912): No heartbeat from core client for 30 sec - exiting 00:20:16 (1912): No heartbeat from core client for 30 sec - exiting 00:20:17 (1912): No heartbeat from core client for 30 sec - exiting 00:20:18 (1912): No heartbeat from core client for 30 sec - exiting 00:20:19 (1912): No heartbeat from core client for 30 sec - exiting 00:20:20 (1912): No heartbeat from core client for 30 sec - exiting 00:20:21 (1912): No heartbeat from core client for 30 sec - exiting 00:20:22 (1912): No heartbeat from core client for 30 sec - exiting 00:20:23 (1912): No heartbeat from core client for 30 sec - exiting 00:20:24 (1912): No heartbeat from core client for 30 sec - exiting 00:20:25 (1912): No heartbeat from core client for 30 sec - exiting 00:20:26 (1912): No heartbeat from core client for 30 sec - exiting 00:20:27 (1912): No heartbeat from core client for 30 sec - exiting 00:20:28 (1912): No heartbeat from core client for 30 sec - exiting 00:20:29 (1912): No heartbeat from core client for 30 sec - exiting 00:20:30 (1912): No heartbeat from core client for 30 sec - exiting 00:20:31 (1912): No heartbeat from core client for 30 sec - exiting 00:20:32 (1912): No heartbeat from core client for 30 sec - exiting 00:20:33 (1912): No heartbeat from core client for 30 sec - exiting 00:20:34 (1912): No heartbeat from core client for 30 sec - exiting 00:20:35 (1912): No heartbeat from core client for 30 sec - exiting 00:20:36 (1912): No heartbeat from core client for 30 sec - exiting 00:20:37 (1912): No heartbeat from core client for 30 sec - exiting 00:20:38 (1912): No heartbeat from core client for 30 sec - exiting 00:20:39 (1912): No heartbeat from core client for 30 sec - exiting 00:20:40 (1912): No heartbeat from core client for 30 sec - exiting 00:20:41 (1912): No heartbeat from core client for 30 sec - exiting 00:20:42 (1912): No heartbeat from core client for 30 sec - exiting 00:20:43 (1912): No heartbeat from core client for 30 sec - exiting 00:20:44 (1912): No heartbeat from core client for 30 sec - exiting 00:20:45 (1912): No heartbeat from core client for 30 sec - exiting 00:20:46 (1912): No heartbeat from core client for 30 sec - exiting 00:20:47 (1912): No heartbeat from core client for 30 sec - exiting 00:20:48 (1912): No heartbeat from core client for 30 sec - exiting 00:20:49 (1912): No heartbeat from core client for 30 sec - exiting 00:20:50 (1912): No heartbeat from core client for 30 sec - exiting 00:20:51 (1912): No heartbeat from core client for 30 sec - exiting 00:20:52 (1912): No heartbeat from core client for 30 sec - exiting 00:20:53 (1912): No heartbeat from core client for 30 sec - exiting 00:20:54 (1912): No heartbeat from core client for 30 sec - exiting 00:20:55 (1912): No heartbeat from core client for 30 sec - exiting 00:20:56 (1912): No heartbeat from core client for 30 sec - exiting 00:20:57 (1912): No heartbeat from core client for 30 sec - exiting 00:20:58 (1912): No heartbeat from core client for 30 sec - exiting 00:20:59 (1912): No heartbeat from core client for 30 sec - exiting 00:21:00 (1912): No heartbeat from core client for 30 sec - exiting 00:21:01 (1912): No heartbeat from core client for 30 sec - exiting 00:21:02 (1912): No heartbeat from core client for 30 sec - exiting 00:21:03 (1912): No heartbeat from core client for 30 sec - exiting 00:21:04 (1912): No heartbeat from core client for 30 sec - exiting 00:21:05 (1912): No heartbeat from core client for 30 sec - exiting 00:21:06 (1912): No heartbeat from core client for 30 sec - exiting 00:21:07 (1912): No heartbeat from core client for 30 sec - exiting 00:21:08 (1912): No heartbeat from core client for 30 sec - exiting 00:21:09 (1912): No heartbeat from core client for 30 sec - exiting 00:21:10 (1912): No heartbeat from core client for 30 sec - exiting 00:21:11 (1912): No heartbeat from core client for 30 sec - exiting 00:21:12 (1912): No heartbeat from core client for 30 sec - exiting 00:21:13 (1912): No heartbeat from core client for 30 sec - exiting 00:21:14 (1912): No heartbeat from core client for 30 sec - exiting 00:21:15 (1912): No heartbeat from core client for 30 sec - exiting 00:21:16 (1912): No heartbeat from core client for 30 sec - exiting 00:21:17 (1912): No heartbeat from core client for 30 sec - exiting 00:21:18 (1912): No heartbeat from core client for 30 sec - exiting 00:21:19 (1912): No heartbeat from core client for 30 sec - exiting 00:21:20 (1912): No heartbeat from core client for 30 sec - exiting 00:21:21 (1912): No heartbeat from core client for 30 sec - exiting 00:21:22 (1912): No heartbeat from core client for 30 sec - exiting 00:21:23 (1912): No heartbeat from core client for 30 sec - exiting 00:21:24 (1912): No heartbeat from core client for 30 sec - exiting 00:21:25 (1912): No heartbeat from core client for 30 sec - exiting 00:21:26 (1912): No heartbeat from core client for 30 sec - exiting 00:21:27 (1912): No heartbeat from core client for 30 sec - exiting 00:21:28 (1912): No heartbeat from core client for 30 sec - exiting 00:21:29 (1912): No heartbeat from core client for 30 sec - exiting 00:21:30 (1912): No heartbeat from core client for 30 sec - exiting 00:21:31 (1912): No heartbeat from core client for 30 sec - exiting 00:21:32 (1912): No heartbeat from core client for 30 sec - exiting 00:21:33 (1912): No heartbeat from core client for 30 sec - exiting 00:21:34 (1912): No heartbeat from core client for 30 sec - exiting 00:21:35 (1912): No heartbeat from core client for 30 sec - exiting 00:21:37 (1912): No heartbeat from core client for 30 sec - exiting 00:21:38 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:21:39 (1912): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t37wko.pje2c10 Error converting file to netcdf: dataout/t37wko.pie2c10 Error converting file to netcdf: dataout/t37wko.pfe2c10 Error converting file to netcdf: dataout/t37wka.phe2c10 Error converting file to netcdf: dataout/t37wka.pge2c10 Error converting file to netcdf: dataout/t37wka.pee2c10 Error converting file to netcdf: dataout/t37wka.pde2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Sep 2011 17:27:52	931768	13396440	hadcm3n_t37w_1940_40_007446063_2	77,760	199,383	2.5641
24 Sep 2011 05:24:36	931768	13396440	hadcm3n_t37w_1940_40_007446063_2	51,840	134,522	2.5949
22 Sep 2011 16:29:14	931768	13396440	hadcm3n_t37w_1940_40_007446063_2	25,920	68,612	2.6471