Task 15544898

Name	hadcm3n_o0n6_2140_40_008269770_2
Workunit	8424894
Created	15 Jan 2013, 17:37:36 UTC
Sent	15 Jan 2013, 17:37:43 UTC
Report deadline	17 Apr 2013, 1:04:54 UTC
Received	5 Apr 2013, 22:17:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	325479
Run time	43 days 18 hours 13 min 52 sec
CPU time	43 days 18 hours 13 min 52 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.2.13</core_client_version> <message>The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:47:41 (1952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:42 (1952): No heartbeat from core client for 30 sec - exiting 08:06:55 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0n6ko.pjy4c10 Error converting file to netcdf: dataout/o0n6ko.piy4c10 Error converting file to netcdf: dataout/o0n6ko.pfy4c10 Error converting file to netcdf: dataout/o0n6ka.phy4c10 Error converting file to netcdf: dataout/o0n6ka.pgy4c10 Error converting file to netcdf: dataout/o0n6ka.pey4c10 Error converting file to netcdf: dataout/o0n6ka.pdy4c10 09:08:15 (4916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:32:45 (3016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:56:45 (5044): No heartbeat from core client for 30 sec - exiting 08:56:46 (5044): No heartbeat from core client for 30 sec - exiting 08:56:47 (5044): No heartbeat from core client for 30 sec - exiting 08:56:48 (5044): No heartbeat from core client for 30 sec - exiting 08:56:49 (5044): No heartbeat from core client for 30 sec - exiting 08:56:50 (5044): No heartbeat from core client for 30 sec - exiting 08:56:51 (5044): No heartbeat from core client for 30 sec - exiting 08:56:52 (5044): No heartbeat from core client for 30 sec - exiting 08:56:53 (5044): No heartbeat from core client for 30 sec - exiting 08:56:55 (5044): No heartbeat from core client for 30 sec - exiting 08:56:56 (5044): No heartbeat from core client for 30 sec - exiting 08:56:57 (5044): No heartbeat from core client for 30 sec - exiting 08:56:58 (5044): No heartbeat from core client for 30 sec - exiting 08:56:59 (5044): No heartbeat from core client for 30 sec - exiting 08:57:00 (5044): No heartbeat from core client for 30 sec - exiting 08:57:01 (5044): No heartbeat from core client for 30 sec - exiting 08:57:02 (5044): No heartbeat from core client for 30 sec - exiting 08:57:03 (5044): No heartbeat from core client for 30 sec - exiting 08:57:04 (5044): No heartbeat from core client for 30 sec - exiting 08:57:06 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:59:07 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:03 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:21:24 PM No files match the supplied pattern. MainError: 09:21:24 PM No files match the supplied pattern. MainError: 02:34:26 PM No files match the supplied pattern. MainError: 02:34:26 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 07:05:49 AM No files match the supplied pattern. MainError: 07:05:49 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1672, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1672, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1672, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... MainError: 12:42:52 AM No files match the supplied pattern. MainError: 12:42:52 AM No files match the supplied pattern. MainError: 05:22:05 AM No files match the supplied pattern. MainError: 05:22:05 AM No files match the supplied pattern. 22:22:40 (2372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 09:28:04 PM No files match the supplied pattern. MainError: 09:28:04 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:50:19 AM No files match the supplied pattern. MainError: 03:50:19 AM No files match the supplied pattern. MainError: 08:08:07 PM No files match the supplied pattern. MainError: 08:08:07 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:35:54 AM No files match the supplied pattern. MainError: 08:35:54 AM No files match the supplied pattern. MainError: 01:18:00 AM No files match the supplied pattern. MainError: 01:18:00 AM No files match the supplied pattern. 12:03:06 (240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:07 (240): No heartbeat from core client for 30 sec - exiting Error converting file to netcdf: dataout/o0n6ka.ph11c10 Error converting file to netcdf: dataout/o0n6ka.pg11c10 Error converting file to netcdf: dataout/o0n6ka.pe11c10 MainError: 06:52:00 PM No files match the supplied pattern. MainError: 06:52:00 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Apr 2013 18:58:46	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	777,600	3,858,950	4.9626
04 Apr 2013 01:22:51	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	751,680	3,724,403	4.9548
02 Apr 2013 08:38:31	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	725,760	3,590,242	4.9469
28 Mar 2013 20:13:17	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	699,840	3,457,023	4.9397
27 Mar 2013 03:55:34	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	673,920	3,324,145	4.9326
22 Mar 2013 21:32:16	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	648,000	3,191,643	4.9254
21 Mar 2013 05:23:23	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	622,080	3,058,735	4.9169
19 Mar 2013 12:46:09	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	596,160	2,925,799	4.9077
15 Mar 2013 07:08:04	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	570,240	2,792,928	4.8978
13 Mar 2013 14:40:22	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	544,320	2,659,574	4.8860
11 Mar 2013 21:27:38	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	518,400	2,524,765	4.8703
07 Mar 2013 15:05:29	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	492,480	2,390,617	4.8542
06 Mar 2013 00:03:46	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	466,560	2,257,155	4.8379
01 Mar 2013 14:45:44	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	440,640	2,123,577	4.8193
27 Feb 2013 22:29:52	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	414,720	1,991,393	4.8018
26 Feb 2013 06:19:33	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	388,800	1,858,612	4.7804
21 Feb 2013 21:13:40	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	362,880	1,726,352	4.7574
20 Feb 2013 04:27:12	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	336,960	1,593,029	4.7277
19 Feb 2013 00:31:24	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	311,040	1,496,009	4.8097
15 Feb 2013 09:37:17	325479	15544898	hadcm3n_o0n6_2140_40_008269770_2	285,120	1,411,004	4.9488