Task 15559284

Name	hadcm3n_o159_2140_40_008269529_3
Workunit	8424653
Created	27 Jan 2013, 12:26:05 UTC
Sent	27 Jan 2013, 12:26:13 UTC
Report deadline	28 Apr 2013, 19:53:24 UTC
Received	16 Feb 2013, 23:38:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1168327
Run time	19 days 16 hours 55 min 33 sec
CPU time	15 days 16 hours 12 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.36 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7804, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o159ko.pjy3c10 Error converting file to netcdf: dataout/o159ko.piy3c10 Error converting file to netcdf: dataout/o159ko.pfy3c10 Error converting file to netcdf: dataout/o159ka.phy3c10 Error converting file to netcdf: dataout/o159ka.pgy3c10 Error converting file to netcdf: dataout/o159ka.pey3c10 Error converting file to netcdf: dataout/o159ka.pdy3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13248, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:53:43 (9816): Can't acquire lockfile (32) - waiting 35s 11:54:08 (10072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:57:57 (6904): Can't acquire lockfile (32) - waiting 35s 11:58:26 (9816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:13:44 PM No files match the supplied pattern. MainError: 11:13:44 PM No files match the supplied pattern. MainError: 02:24:20 PM No files match the supplied pattern. MainError: 02:24:20 PM No files match the supplied pattern. MainError: 05:32:30 AM No files match the supplied pattern. MainError: 05:32:30 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:35:44 PM No files match the supplied pattern. MainError: 09:35:44 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... MainError: 03:45:49 PM No files match the supplied pattern. MainError: 03:45:49 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 09:05:55 AM No files match the supplied pattern. MainError: 09:05:55 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:31:48 (5444): No heartbeat from core client for 30 sec - exiting 17:31:49 (5444): No heartbeat from core client for 30 sec - exiting 17:31:50 (5444): No heartbeat from core client for 30 sec - exiting 17:31:51 (5444): No heartbeat from core client for 30 sec - exiting 17:31:52 (5444): No heartbeat from core client for 30 sec - exiting 17:31:53 (5444): No heartbeat from core client for 30 sec - exiting 17:31:54 (5444): No heartbeat from core client for 30 sec - exiting 17:31:55 (5444): No heartbeat from core client for 30 sec - exiting 17:31:56 (5444): No heartbeat from core client for 30 sec - exiting 17:31:57 (5444): No heartbeat from core client for 30 sec - exiting 17:31:58 (5444): No heartbeat from core client for 30 sec - exiting 17:31:59 (5444): No heartbeat from core client for 30 sec - exiting 17:32:00 (5444): No heartbeat from core client for 30 sec - exiting 17:32:01 (5444): No heartbeat from core client for 30 sec - exiting 17:32:02 (5444): No heartbeat from core client for 30 sec - exiting 17:32:03 (5444): No heartbeat from core client for 30 sec - exiting 17:32:04 (5444): No heartbeat from core client for 30 sec - exiting 17:32:05 (5444): No heartbeat from core client for 30 sec - exiting 17:32:06 (5444): No heartbeat from core client for 30 sec - exiting 17:32:07 (5444): No heartbeat from core client for 30 sec - exiting 17:32:08 (5444): No heartbeat from core client for 30 sec - exiting 17:32:09 (5444): No heartbeat from core client for 30 sec - exiting 17:32:10 (5444): No heartbeat from core client for 30 sec - exiting 17:32:11 (5444): No heartbeat from core client for 30 sec - exiting 17:32:12 (5444): No heartbeat from core client for 30 sec - exiting 17:32:13 (5444): No heartbeat from core client for 30 sec - exiting 17:32:14 (5444): No heartbeat from core client for 30 sec - exiting 17:32:15 (5444): No heartbeat from core client for 30 sec - exiting 17:32:16 (5444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:32:18 (5444): No heartbeat from core client for 30 sec - exiting MainError: 02:13:04 AM No files match the supplied pattern. MainError: 02:13:04 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 07:07:51 PM No files match the supplied pattern. MainError: 07:07:51 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 11:37:25 AM No files match the supplied pattern. MainError: 11:37:25 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 05:04:23 AM No files match the supplied pattern. MainError: 05:04:23 AM No files match the supplied pattern. Error converting file to netcdf: dataout/o159ka.ph11c10 Error converting file to netcdf: dataout/o159ka.pg11c10 Error converting file to netcdf: dataout/o159ka.pe11c10 MainError: 09:35:40 PM No files match the supplied pattern. MainError: 09:35:40 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Feb 2013 21:39:12	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	777,600	1,429,557	1.8384
16 Feb 2013 05:08:12	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	751,680	1,380,655	1.8368
15 Feb 2013 11:37:40	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	725,760	1,332,140	1.8355
14 Feb 2013 19:18:00	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	699,840	1,283,851	1.8345
14 Feb 2013 02:19:18	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	673,920	1,235,159	1.8328
13 Feb 2013 09:06:26	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	648,000	1,187,136	1.8320
12 Feb 2013 15:46:21	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	622,080	1,138,524	1.8302
11 Feb 2013 21:41:28	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	596,160	1,088,275	1.8255
11 Feb 2013 05:35:30	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	570,240	1,040,618	1.8249
10 Feb 2013 14:25:35	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	544,320	993,789	1.8257
09 Feb 2013 23:14:36	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	518,400	946,774	1.8263
09 Feb 2013 08:46:47	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	492,480	899,960	1.8274
08 Feb 2013 16:15:56	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	466,560	852,475	1.8271
08 Feb 2013 00:18:43	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	440,640	804,522	1.8258
07 Feb 2013 07:02:17	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	414,720	755,637	1.8220
06 Feb 2013 15:10:43	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	388,800	707,645	1.8201
06 Feb 2013 01:11:27	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	362,880	661,779	1.8237
05 Feb 2013 06:01:21	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	336,960	613,741	1.8214
04 Feb 2013 12:35:06	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	311,040	566,059	1.8199
03 Feb 2013 22:35:08	1168327	15559284	hadcm3n_o159_2140_40_008269529_3	285,120	519,241	1.8211