Task 13550076

Name	hadcm3n_y8oc_1900_40_007522192_0
Workunit	7719667
Created	28 Oct 2011, 13:16:39 UTC
Sent	1 Nov 2011, 19:36:33 UTC
Report deadline	1 Feb 2012, 3:03:44 UTC
Received	15 Nov 2011, 17:18:22 UTC
Server state	Over
Outcome	Computation error
Client state	Done
Exit status	22 (0x00000016) Unknown error code
Computer ID	270656
Run time	3 days 10 hours 25 min 9 sec
CPU time	3 days 10 hours 25 min 9 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	1.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.10.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 19:14:32 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:33 (3944): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 05:56:19 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:56:20 (3960): No heartbeat from core client for 30 sec - exiting 05:56:21 (3960): No heartbeat from core client for 30 sec - exiting 05:56:22 (3960): No heartbeat from core client for 30 sec - exiting 05:56:23 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:25:46 (2808): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:25:47 (2808): No heartbeat from core client for 30 sec - exiting 06:18:28 (1380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:29 (1380): No heartbeat from core client for 30 sec - exiting 06:18:30 (1380): No heartbeat from core client for 30 sec - exiting 06:18:32 (1380): No heartbeat from core client for 30 sec - exiting 10:21:58 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:59 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:44:56 (1892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y8ocko.pja1c10 Error converting file to netcdf: dataout/y8ocko.pia1c10 Error converting file to netcdf: dataout/y8ocko.pfa1c10 Error converting file to netcdf: dataout/y8ocka.pha1c10 Error converting file to netcdf: dataout/y8ocka.pga1c10 Error converting file to netcdf: dataout/y8ocka.pea1c10 Error converting file to netcdf: dataout/y8ocka.pda1c10 Suspended CPDN Monitor - Suspend request from BOINC... 11:09:23 (3768): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 11:11:13 (3668): No heartbeat from core client for 30 sec - exiting 11:11:14 (3668): No heartbeat from core client for 30 sec - exiting 11:11:15 (3668): No heartbeat from core client for 30 sec - exiting 11:11:16 (3668): No heartbeat from core client for 30 sec - exiting 11:11:17 (3668): No heartbeat from core client for 30 sec - exiting 11:11:18 (3668): No heartbeat from core client for 30 sec - exiting 11:11:20 (3668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:06:16 (2720): No heartbeat from core client for 30 sec - exiting 04:06:18 (2720): No heartbeat from core client for 30 sec - exiting 04:06:19 (2720): No heartbeat from core client for 30 sec - exiting 04:06:21 (2720): No heartbeat from core client for 30 sec - exiting 04:06:22 (2720): No heartbeat from core client for 30 sec - exiting 04:06:23 (2720): No heartbeat from core client for 30 sec - exiting 04:06:24 (2720): No heartbeat from core client for 30 sec - exiting 04:06:25 (2720): No heartbeat from core client for 30 sec - exiting 04:06:26 (2720): No heartbeat from core client for 30 sec - exiting 04:06:27 (2720): No heartbeat from core client for 30 sec - exiting 04:06:28 (2720): No heartbeat from core client for 30 sec - exiting 04:06:29 (2720): No heartbeat from core client for 30 sec - exiting 04:06:30 (2720): No heartbeat from core client for 30 sec - exiting 04:06:31 (2720): No heartbeat from core client for 30 sec - exiting 04:06:33 (2720): No heartbeat from core client for 30 sec - exiting 04:06:34 (2720): No heartbeat from core client for 30 sec - exiting 04:06:35 (2720): No heartbeat from core client for 30 sec - exiting 04:06:36 (2720): No heartbeat from core client for 30 sec - exiting 04:06:37 (2720): No heartbeat from core client for 30 sec - exiting 04:06:38 (2720): No heartbeat from core client for 30 sec - exiting 04:06:39 (2720): No heartbeat from core client for 30 sec - exiting 04:06:40 (2720): No heartbeat from core client for 30 sec - exiting 04:06:41 (2720): No heartbeat from core client for 30 sec - exiting 04:06:42 (2720): No heartbeat from core client for 30 sec - exiting 04:06:44 (2720): No heartbeat from core client for 30 sec - exiting 04:06:45 (2720): No heartbeat from core client for 30 sec - exiting 04:06:46 (2720): No heartbeat from core client for 30 sec - exiting 04:06:47 (2720): No heartbeat from core client for 30 sec - exiting 04:06:48 (2720): No heartbeat from core client for 30 sec - exiting 04:06:49 (2720): No heartbeat from core client for 30 sec - exiting 04:06:50 (2720): No heartbeat from core client for 30 sec - exiting 04:06:51 (2720): No heartbeat from core client for 30 sec - exiting 04:06:52 (2720): No heartbeat from core client for 30 sec - exiting 04:06:53 (2720): No heartbeat from core client for 30 sec - exiting 04:06:54 (2720): No heartbeat from core client for 30 sec - exiting 04:06:56 (2720): No heartbeat from core client for 30 sec - exiting 04:06:57 (2720): No heartbeat from core client for 30 sec - exiting 04:06:58 (2720): No heartbeat from core client for 30 sec - exiting 04:06:59 (2720): No heartbeat from core client for 30 sec - exiting 04:07:00 (2720): No heartbeat from core client for 30 sec - exiting 04:07:01 (2720): No heartbeat from core client for 30 sec - exiting 04:07:02 (2720): No heartbeat from core client for 30 sec - exiting 04:07:03 (2720): No heartbeat from core client for 30 sec - exiting 04:07:04 (2720): No heartbeat from core client for 30 sec - exiting 04:07:05 (2720): No heartbeat from core client for 30 sec - exiting 04:07:06 (2720): No heartbeat from core client for 30 sec - exiting 04:07:08 (2720): No heartbeat from core client for 30 sec - exiting 04:07:09 (2720): No heartbeat from core client for 30 sec - exiting 04:07:10 (2720): No heartbeat from core client for 30 sec - exiting 04:07:11 (2720): No heartbeat from core client for 30 sec - exiting 04:07:12 (2720): No heartbeat from core client for 30 sec - exiting 04:07:13 (2720): No heartbeat from core client for 30 sec - exiting 04:07:14 (2720): No heartbeat from core client for 30 sec - exiting 04:07:15 (2720): No heartbeat from core client for 30 sec - exiting 04:07:16 (2720): No heartbeat from core client for 30 sec - exiting 04:07:17 (2720): No heartbeat from core client for 30 sec - exiting 04:07:19 (2720): No heartbeat from core client for 30 sec - exiting 04:07:20 (2720): No heartbeat from core client for 30 sec - exiting 04:07:21 (2720): No heartbeat from core client for 30 sec - exiting 04:07:22 (2720): No heartbeat from core client for 30 sec - exiting 04:07:23 (2720): No heartbeat from core client for 30 sec - exiting 04:07:24 (2720): No heartbeat from core client for 30 sec - exiting 04:07:25 (2720): No heartbeat from core client for 30 sec - exiting 04:07:26 (2720): No heartbeat from core client for 30 sec - exiting 04:07:27 (2720): No heartbeat from core client for 30 sec - exiting 04:07:28 (2720): No heartbeat from core client for 30 sec - exiting 04:07:30 (2720): No heartbeat from core client for 30 sec - exiting 04:07:31 (2720): No heartbeat from core client for 30 sec - exiting 04:07:32 (2720): No heartbeat from core client for 30 sec - exiting 04:07:33 (2720): No heartbeat from core client for 30 sec - exiting 04:07:34 (2720): No heartbeat from core client for 30 sec - exiting 04:07:35 (2720): No heartbeat from core client for 30 sec - exiting 04:07:36 (2720): No heartbeat from core client for 30 sec - exiting 04:07:38 (2720): No heartbeat from core client for 30 sec - exiting 04:07:39 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:00:04 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:29 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:30 (3100): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:16:40 (4380): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:16:42 (4380): No heartbeat from core client for 30 sec - exiting 19:16:43 (4380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 07:25:24 (3400): No heartbeat from core client for 30 sec - exiting 07:25:25 (3400): No heartbeat from core client for 30 sec - exiting 07:25:26 (3400): No heartbeat from core client for 30 sec - exiting 07:25:27 (3400): No heartbeat from core client for 30 sec - exiting 07:25:28 (3400): No heartbeat from core client for 30 sec - exiting 07:25:29 (3400): No heartbeat from core client for 30 sec - exiting 07:25:30 (3400): No heartbeat from core client for 30 sec - exiting 07:25:31 (3400): No heartbeat from core client for 30 sec - exiting 07:25:32 (3400): No heartbeat from core client for 30 sec - exiting 07:25:33 (3400): No heartbeat from core client for 30 sec - exiting 07:25:35 (3400): No heartbeat from core client for 30 sec - exiting 07:25:36 (3400): No heartbeat from core client for 30 sec - exiting 07:25:37 (3400): No heartbeat from core client for 30 sec - exiting 07:25:38 (3400): No heartbeat from core client for 30 sec - exiting 07:25:39 (3400): No heartbeat from core client for 30 sec - exiting 07:25:40 (3400): No heartbeat from core client for 30 sec - exiting 07:25:41 (3400): No heartbeat from core client for 30 sec - exiting 07:25:42 (3400): No heartbeat from core client for 30 sec - exiting 07:25:43 (3400): No heartbeat from core client for 30 sec - exiting 07:25:44 (3400): No heartbeat from core client for 30 sec - exiting 07:25:45 (3400): No heartbeat from core client for 30 sec - exiting 07:25:47 (3400): No heartbeat from core client for 30 sec - exiting 07:25:48 (3400): No heartbeat from core client for 30 sec - exiting 07:25:49 (3400): No heartbeat from core client for 30 sec - exiting 07:25:50 (3400): No heartbeat from core client for 30 sec - exiting 07:25:51 (3400): No heartbeat from core client for 30 sec - exiting 07:25:52 (3400): No heartbeat from core client for 30 sec - exiting 07:25:53 (3400): No heartbeat from core client for 30 sec - exiting 07:25:54 (3400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:08:20 (180): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:22:50 (3624): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 11:22:51 (3624): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2480, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2480, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2480, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2480, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3228, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3228, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 16:27:41 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Nov 2011 18:50:12	270656	13550076	hadcm3n_y8oc_1900_40_007522192_0	51,840	227,429	4.3871
05 Nov 2011 02:45:55	270656	13550076	hadcm3n_y8oc_1900_40_007522192_0	25,920	114,624	4.4222