Task 11502922

Name	famous_udk6_599_200_006650417_4
Workunit	6853789
Created	10 Jun 2010, 13:33:56 UTC
Sent	15 Aug 2010, 21:09:40 UTC
Report deadline	15 Nov 2010, 4:36:51 UTC
Received	3 Oct 2010, 6:18:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	849720
Run time	25 days 7 hours 9 min 42 sec
CPU time	10 days 3 hours 2 min 55 sec
Validate state	Invalid
Credit	4,076.46
Device peak FLOPS	1.95 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4988, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:08:40 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:57 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:15 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:01 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:07 (4772): No heartbeat from core client for 30 sec - exiting 15:22:42 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:27 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:28 (4984): No heartbeat from core client for 30 sec - exiting 21:46:29 (4984): No heartbeat from core client for 30 sec - exiting 03:47:54 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:55 (5000): No heartbeat from core client for 30 sec - exiting 03:47:56 (5000): No heartbeat from core client for 30 sec - exiting 03:47:57 (5000): No heartbeat from core client for 30 sec - exiting 03:47:58 (5000): No heartbeat from core client for 30 sec - exiting 05:52:30 (6624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:31 (6624): No heartbeat from core client for 30 sec - exiting 05:52:32 (6624): No heartbeat from core client for 30 sec - exiting 05:52:33 (6624): No heartbeat from core client for 30 sec - exiting 05:52:34 (6624): No heartbeat from core client for 30 sec - exiting 05:52:35 (6624): No heartbeat from core client for 30 sec - exiting 05:52:36 (6624): No heartbeat from core client for 30 sec - exiting 08:00:01 (5512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:02 (5512): No heartbeat from core client for 30 sec - exiting 10:10:55 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:05 (612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:44 (4440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:45 (4440): No heartbeat from core client for 30 sec - exiting 04:18:34 (7028): No heartbeat from core client for 30 sec - exiting 04:18:35 (7028): No heartbeat from core client for 30 sec - exiting 04:18:39 (7028): No heartbeat from core client for 30 sec - exiting 04:18:40 (7028): No heartbeat from core client for 30 sec - exiting 04:18:41 (7028): No heartbeat from core client for 30 sec - exiting 04:18:42 (7028): No heartbeat from core client for 30 sec - exiting 04:18:43 (7028): No heartbeat from core client for 30 sec - exiting 04:18:44 (7028): No heartbeat from core client for 30 sec - exiting 04:18:45 (7028): No heartbeat from core client for 30 sec - exiting 04:18:46 (7028): No heartbeat from core client for 30 sec - exiting 04:18:47 (7028): No heartbeat from core client for 30 sec - exiting 04:18:48 (7028): No heartbeat from core client for 30 sec - exiting 04:18:49 (7028): No heartbeat from core client for 30 sec - exiting 04:18:50 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:55 (6248): Can't acquire lockfile (32) - waiting 35s BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 10:04:58 (3408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:25:06 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:08 (6132): No heartbeat from core client for 30 sec - exiting 07:22:26 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:17:12 (336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:49 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:04 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3112, iMonCtr=1 Model crash detected, will try to restart... 03:34:58 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:00 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:06 (5764): No heartbeat from core client for 30 sec - exiting 06:36:07 (5764): No heartbeat from core client for 30 sec - exiting 06:36:08 (5764): No heartbeat from core client for 30 sec - exiting 06:36:09 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:39 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:59:56 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:54 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:15:15 (10300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:51 (20372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:52 (20372): No heartbeat from core client for 30 sec - exiting 20:41:53 (20372): No heartbeat from core client for 30 sec - exiting 20:41:54 (20372): No heartbeat from core client for 30 sec - exiting 20:41:55 (20372): No heartbeat from core client for 30 sec - exiting 20:41:56 (20372): No heartbeat from core client for 30 sec - exiting 20:41:57 (20372): No heartbeat from core client for 30 sec - exiting 20:41:58 (20372): No heartbeat from core client for 30 sec - exiting 20:41:59 (20372): No heartbeat from core client for 30 sec - exiting 20:42:00 (20372): No heartbeat from core client for 30 sec - exiting 20:42:01 (20372): No heartbeat from core client for 30 sec - exiting 23:00:21 (22044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:22 (22044): No heartbeat from core client for 30 sec - exiting 23:00:23 (22044): No heartbeat from core client for 30 sec - exiting 23:00:24 (22044): No heartbeat from core client for 30 sec - exiting 01:42:18 (22168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:19 (22168): No heartbeat from core client for 30 sec - exiting 01:42:20 (22168): No heartbeat from core client for 30 sec - exiting 01:42:21 (22168): No heartbeat from core client for 30 sec - exiting 06:34:39 (24328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=22964, selfPID=22964, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( 00:38:53 (29404): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,235,546	869,756	0.7039
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,226,186	863,297	0.7041
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,216,826	856,682	0.7040
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,207,466	850,341	0.7042
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,198,106	843,916	0.7044
03 Oct 2010 06:20:11	849720	11502922	famous_udk6_599_200_006650417_4	1,188,746	837,259	0.7043
01 Oct 2010 12:01:50	849720	11502922	famous_udk6_599_200_006650417_4	1,179,386	829,547	0.7034
01 Oct 2010 05:18:11	849720	11502922	famous_udk6_599_200_006650417_4	1,170,026	822,206	0.7027
01 Oct 2010 03:23:54	849720	11502922	famous_udk6_599_200_006650417_4	1,160,666	815,916	0.7030
30 Sep 2010 20:41:03	849720	11502922	famous_udk6_599_200_006650417_4	1,151,306	809,313	0.7030
30 Sep 2010 15:53:11	849720	11502922	famous_udk6_599_200_006650417_4	1,141,946	802,771	0.7030
30 Sep 2010 11:49:34	849720	11502922	famous_udk6_599_200_006650417_4	1,132,586	796,483	0.7032
30 Sep 2010 07:27:41	849720	11502922	famous_udk6_599_200_006650417_4	1,123,226	789,990	0.7033
30 Sep 2010 03:20:56	849720	11502922	famous_udk6_599_200_006650417_4	1,113,866	783,628	0.7035
29 Sep 2010 22:46:22	849720	11502922	famous_udk6_599_200_006650417_4	1,104,506	777,001	0.7035
29 Sep 2010 18:25:03	849720	11502922	famous_udk6_599_200_006650417_4	1,095,146	770,535	0.7036
29 Sep 2010 13:39:20	849720	11502922	famous_udk6_599_200_006650417_4	1,085,786	763,891	0.7035
29 Sep 2010 11:09:46	849720	11502922	famous_udk6_599_200_006650417_4	1,076,426	757,345	0.7036
29 Sep 2010 08:44:00	849720	11502922	famous_udk6_599_200_006650417_4	1,067,066	750,583	0.7034
29 Sep 2010 06:12:24	849720	11502922	famous_udk6_599_200_006650417_4	1,057,706	743,858	0.7033