Task 11702020

Name	famous_v22k_1599_200_006688450_1
Workunit	6891703
Created	26 Aug 2010, 15:47:53 UTC
Sent	4 Sep 2010, 2:10:12 UTC
Report deadline	4 Dec 2010, 9:37:23 UTC
Received	28 Nov 2010, 6:44:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1058331
Run time	4 days 7 hours 26 min 12 sec
CPU time	3 days 20 hours 15 min 38 sec
Validate state	Invalid
Credit	1,945.63
Device peak FLOPS	2.46 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7148, selfPID=7148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6288, selfPID=6288, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6520, selfPID=6520, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6916, selfPID=6916, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6980, selfPID=6980, iMonCtr=1 12:32:49 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:50 (4172): No heartbeat from core client for 30 sec - exiting 12:32:51 (4172): No heartbeat from core client for 30 sec - exiting 12:32:52 (4172): No heartbeat from core client for 30 sec - exiting 12:32:53 (4172): No heartbeat from core client for 30 sec - exiting 12:32:54 (4172): No heartbeat from core client for 30 sec - exiting 12:32:55 (4172): No heartbeat from core client for 30 sec - exiting 12:32:56 (4172): No heartbeat from core client for 30 sec - exiting 12:32:57 (4172): No heartbeat from core client for 30 sec - exiting 12:32:58 (4172): No heartbeat from core client for 30 sec - exiting 12:32:59 (4172): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6604, selfPID=6604, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=108, selfPID=108, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2032, selfPID=2032, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4500, selfPID=4500, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5564, selfPID=5564, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=396, selfPID=396, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1200, selfPID=1200, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=240, selfPID=240, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6688, selfPID=6688, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5024, selfPID=5024, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5164, selfPID=5164, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4120, selfPID=4120, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2148, selfPID=2148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5564, selfPID=5564, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6328, selfPID=6328, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5792, selfPID=5792, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6488, selfPID=6488, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6984, selfPID=6984, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6152, selfPID=6152, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6676, selfPID=6676, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5656, selfPID=5656, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5060, selfPID=5060, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6332, selfPID=6332, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6776, selfPID=6776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=580, selfPID=580, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1124, selfPID=1124, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6308, selfPID=6308, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1832, selfPID=1832, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3624, selfPID=3624, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7456, selfPID=7456, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5060, selfPID=5060, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7764, selfPID=7764, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6488, selfPID=6488, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7144, selfPID=7144, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5956, selfPID=5956, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4712, selfPID=4712, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5792, selfPID=5792, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1008, selfPID=1008, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7936, selfPID=7936, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6536, selfPID=6536, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7728, selfPID=7728, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3144, selfPID=3144, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4992, selfPID=4992, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8028, selfPID=8028, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5404, selfPID=5404, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4500, selfPID=4500, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3936, selfPID=3936, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4076, selfPID=4076, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2520, selfPID=2520, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3576, selfPID=3576, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6624, selfPID=6624, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4188, selfPID=4188, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7116, selfPID=7116, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6288, selfPID=6288, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2736, selfPID=2736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6480, selfPID=6480, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3380, selfPID=3380, iMonCtr=1 07:05:06 (6116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:07 (6116): No heartbeat from core client for 30 sec - exiting 07:05:08 (6116): No heartbeat from core client for 30 sec - exiting 07:05:09 (6116): No heartbeat from core client for 30 sec - exiting 07:05:10 (6116): No heartbeat from core client for 30 sec - exiting 07:05:11 (6116): No heartbeat from core client for 30 sec - exiting 07:05:12 (6116): No heartbeat from core client for 30 sec - exiting 07:05:13 (6116): No heartbeat from core client for 30 sec - exiting 07:05:14 (6116): No heartbeat from core client for 30 sec - exiting 07:05:15 (6116): No heartbeat from core client for 30 sec - exiting 07:05:16 (6116): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=612, selfPID=612, iMonCtr=1 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 09:56:35 (3296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:36 (3296): No heartbeat from core client for 30 sec - exiting 09:56:37 (3296): No heartbeat from core client for 30 sec - exiting 09:56:38 (3296): No heartbeat from core client for 30 sec - exiting 09:56:39 (3296): No heartbeat from core client for 30 sec - exiting 09:56:40 (3296): No heartbeat from core client for 30 sec - exiting 09:56:42 (3296): No heartbeat from core client for 30 sec - exiting 09:56:43 (3296): No heartbeat from core client for 30 sec - exiting 09:56:44 (3296): No heartbeat from core client for 30 sec - exiting 09:56:45 (3296): No heartbeat from core client for 30 sec - exiting 09:56:46 (3296): No heartbeat from core client for 30 sec - exiting 09:56:47 (3296): No heartbeat from core client for 30 sec - exiting 09:56:48 (3296): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4724, selfPID=4724, iMonCtr=1 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy no start tag in app init data 10:02:35 (5808): Can't parse init data file - running in standalone mode no start tag in app init data 10:02:35 (3940): Can't parse init data file - running in standalone mode 10:03:55 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:56 (5808): No heartbeat from core client for 30 sec - exiting 10:03:57 (5808): No heartbeat from core client for 30 sec - exiting 10:03:58 (5808): No heartbeat from core client for 30 sec - exiting 10:03:59 (5808): No heartbeat from core client for 30 sec - exiting 10:04:00 (5808): No heartbeat from core client for 30 sec - exiting 10:04:01 (5808): No heartbeat from core client for 30 sec - exiting 10:04:03 (5808): No heartbeat from core client for 30 sec - exiting 10:04:04 (5808): No heartbeat from core client for 30 sec - exiting 10:04:05 (5808): No heartbeat from core client for 30 sec - exiting 10:04:06 (5808): No heartbeat from core client for 30 sec - exiting 10:04:07 (5808): No heartbeat from core client for 30 sec - exiting 10:04:08 (5808): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3940, selfPID=3940, iMonCtr=1 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy no start tag in app init data 10:07:58 (5008): Can't parse init data file - running in standalone mode no start tag in app init data 10:07:58 (1076): Can't parse init data file - running in standaCPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5656, selfPID=5656, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=264, selfPID=264, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3472, selfPID=3472, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4304, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... 19:49:04 (4104): called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4304, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... 19:57:05 (4724): called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4304, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( 01:03:57 (4380): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Nov 2010 06:46:51	1058331	11702020	famous_v22k_1599_200_006688450_1	589,706	328,792	0.5576
28 Nov 2010 06:46:51	1058331	11702020	famous_v22k_1599_200_006688450_1	580,346	323,858	0.5580
28 Nov 2010 06:46:51	1058331	11702020	famous_v22k_1599_200_006688450_1	570,986	318,763	0.5583
28 Nov 2010 00:15:48	1058331	11702020	famous_v22k_1599_200_006688450_1	561,626	313,702	0.5586
27 Nov 2010 22:09:26	1058331	11702020	famous_v22k_1599_200_006688450_1	552,266	308,378	0.5584
27 Nov 2010 18:51:15	1058331	11702020	famous_v22k_1599_200_006688450_1	542,906	303,130	0.5583
27 Nov 2010 17:12:08	1058331	11702020	famous_v22k_1599_200_006688450_1	533,546	298,027	0.5586
27 Nov 2010 11:52:47	1058331	11702020	famous_v22k_1599_200_006688450_1	524,186	292,205	0.5574
27 Nov 2010 11:26:39	1058331	11702020	famous_v22k_1599_200_006688450_1	514,826	287,284	0.5580
27 Nov 2010 11:26:39	1058331	11702020	famous_v22k_1599_200_006688450_1	505,466	282,407	0.5587
19 Sep 2010 22:36:39	1058331	11702020	famous_v22k_1599_200_006688450_1	496,106	277,450	0.5593
19 Sep 2010 20:56:21	1058331	11702020	famous_v22k_1599_200_006688450_1	486,746	272,419	0.5597
19 Sep 2010 17:10:46	1058331	11702020	famous_v22k_1599_200_006688450_1	477,386	267,372	0.5601
15 Sep 2010 02:06:55	1058331	11702020	famous_v22k_1599_200_006688450_1	468,026	261,697	0.5592
14 Sep 2010 23:18:22	1058331	11702020	famous_v22k_1599_200_006688450_1	458,666	256,293	0.5588
14 Sep 2010 21:26:00	1058331	11702020	famous_v22k_1599_200_006688450_1	449,306	250,970	0.5586
14 Sep 2010 19:49:58	1058331	11702020	famous_v22k_1599_200_006688450_1	439,946	245,640	0.5583
14 Sep 2010 18:11:37	1058331	11702020	famous_v22k_1599_200_006688450_1	430,586	240,348	0.5582
14 Sep 2010 16:33:31	1058331	11702020	famous_v22k_1599_200_006688450_1	421,226	235,000	0.5579
14 Sep 2010 14:54:36	1058331	11702020	famous_v22k_1599_200_006688450_1	411,866	229,697	0.5577