Task 15460226

Name	hadcm3n_o1km_2020_40_008244715_3
Workunit	8399839
Created	24 Nov 2012, 18:46:01 UTC
Sent	24 Nov 2012, 18:46:04 UTC
Report deadline	24 Feb 2013, 2:13:15 UTC
Received	17 Dec 2012, 8:50:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	866365
Run time	9 days 18 hours 10 min 22 sec
CPU time	6 days 13 hours 33 min 40 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:12:28 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:30 (4804): No heartbeat from core client for 30 sec - exiting 13:12:31 (4804): No heartbeat from core client for 30 sec - exiting 13:12:32 (4804): No heartbeat from core client for 30 sec - exiting 13:12:33 (4804): No heartbeat from core client for 30 sec - exiting 13:12:34 (4804): No heartbeat from core client for 30 sec - exiting 13:12:35 (4804): No heartbeat from core client for 30 sec - exiting 13:12:36 (4804): No heartbeat from core client for 30 sec - exiting 13:12:37 (4804): No heartbeat from core client for 30 sec - exiting 13:12:38 (4804): No heartbeat from core client for 30 sec - exiting 13:12:40 (4804): No heartbeat from core client for 30 sec - exiting 13:12:41 (4804): No heartbeat from core client for 30 sec - exiting 13:12:42 (4804): No heartbeat from core client for 30 sec - exiting 13:12:43 (4804): No heartbeat from core client for 30 sec - exiting 13:12:44 (4804): No heartbeat from core client for 30 sec - exiting 13:12:45 (4804): No heartbeat from core client for 30 sec - exiting 13:12:46 (4804): No heartbeat from core client for 30 sec - exiting 13:12:47 (4804): No heartbeat from core client for 30 sec - exiting 13:12:48 (4804): No heartbeat from core client for 30 sec - exiting 13:12:49 (4804): No heartbeat from core client for 30 sec - exiting 13:12:51 (4804): No heartbeat from core client for 30 sec - exiting 13:12:52 (4804): No heartbeat from core client for 30 sec - exiting 13:12:53 (4804): No heartbeat from core client for 30 sec - exiting 13:12:54 (4804): No heartbeat from core client for 30 sec - exiting 13:12:55 (4804): No heartbeat from core client for 30 sec - exiting 13:12:56 (4804): No heartbeat from core client for 30 sec - exiting 13:12:57 (4804): No heartbeat from core client for 30 sec - exiting 13:12:58 (4804): No heartbeat from core client for 30 sec - exiting 13:12:59 (4804): No heartbeat from core client for 30 sec - exiting 13:13:00 (4804): No heartbeat from core client for 30 sec - exiting 13:13:01 (4804): No heartbeat from core client for 30 sec - exiting 13:13:03 (4804): No heartbeat from core client for 30 sec - exiting 13:13:04 (4804): No heartbeat from core client for 30 sec - exiting 13:13:05 (4804): No heartbeat from core client for 30 sec - exiting 13:13:06 (4804): No heartbeat from core client for 30 sec - exiting 13:13:07 (4804): No heartbeat from core client for 30 sec - exiting 13:13:08 (4804): No heartbeat from core client for 30 sec - exiting 13:13:09 (4804): No heartbeat from core client for 30 sec - exiting 13:13:10 (4804): No heartbeat from core client for 30 sec - exiting 13:13:11 (4804): No heartbeat from core client for 30 sec - exiting 13:13:12 (4804): No heartbeat from core client for 30 sec - exiting 13:13:13 (4804): No heartbeat from core client for 30 sec - exiting 13:13:15 (4804): No heartbeat from core client for 30 sec - exiting 13:13:16 (4804): No heartbeat from core client for 30 sec - exiting 13:13:17 (4804): No heartbeat from core client for 30 sec - exiting 13:13:18 (4804): No heartbeat from core client for 30 sec - exiting 13:13:19 (4804): No heartbeat from core client for 30 sec - exiting 13:13:20 (4804): No heartbeat from core client for 30 sec - exiting 13:13:21 (4804): No heartbeat from core client for 30 sec - exiting 13:13:22 (4804): No heartbeat from core client for 30 sec - exiting 13:13:23 (4804): No heartbeat from core client for 30 sec - exiting 13:13:24 (4804): No heartbeat from core client for 30 sec - exiting 13:13:26 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:00:18 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:28 (5700): No heartbeat from core client for 30 sec - exiting 14:00:29 (5700): No heartbeat from core client for 30 sec - exiting 14:00:30 (5700): No heartbeat from core client for 30 sec - exiting 14:00:31 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8956, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:45:15 (10096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o1kmko.pjm9c10 Error converting file to netcdf: dataout/o1kmko.pim9c10 Error converting file to netcdf: dataout/o1kmko.pfm9c10 Error converting file to netcdf: dataout/o1kmka.phm9c10 Error converting file to netcdf: dataout/o1kmka.pgm9c10 Error converting file to netcdf: dataout/o1kmka.pem9c10 Error converting file to netcdf: dataout/o1kmka.pdm9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:22:49 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2012 12:00:06	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	233,280	538,044	2.3064
15 Dec 2012 12:00:06	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	207,360	477,479	2.3027
15 Dec 2012 12:00:06	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	181,440	421,649	2.3239
15 Dec 2012 12:00:06	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	155,520	362,742	2.3324
05 Dec 2012 17:06:16	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	129,600	303,034	2.3382
03 Dec 2012 15:10:47	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	103,680	244,000	2.3534
01 Dec 2012 13:52:26	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	77,760	185,868	2.3903
28 Nov 2012 17:08:54	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	51,840	125,658	2.4240
26 Nov 2012 22:01:13	866365	15460226	hadcm3n_o1km_2020_40_008244715_3	25,920	62,677	2.4181