climateprediction.net home page
Task 11690857

Task 11690857

Name famous_v0cb_999_200_006686217_3
Workunit 6889470
Created 26 Aug 2010, 15:39:43 UTC
Sent 29 Aug 2010, 14:35:38 UTC
Report deadline 28 Nov 2010, 22:02:49 UTC
Received 1 Oct 2010, 10:02:54 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1042170
Run time 7 days 19 hours 28 min 23 sec
CPU time 7 days 8 hours 54 min 53 sec
Validate state Invalid
Credit 4,230.87
Device peak FLOPS 2.54 GFLOPS
Application version UK Met Office FAMOUS v6.11
windows_intelx86
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2404, selfPID=2404, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4220, selfPID=4220, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6964, selfPID=6964, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4104, selfPID=4104, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5152, selfPID=5152, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6052, selfPID=6052, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1832, selfPID=1832, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4840, selfPID=4840, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6204, selfPID=6204, iMonCtr=1
07:14:25 (6744): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:14:26 (6744): No heartbeat from core client for 30 sec - exiting
07:14:27 (6744): No heartbeat from core client for 30 sec - exiting
07:14:28 (6744): No heartbeat from core client for 30 sec - exiting
07:14:29 (6744): No heartbeat from core client for 30 sec - exiting
07:14:30 (6744): No heartbeat from core client for 30 sec - exiting
07:14:31 (6744): No heartbeat from core client for 30 sec - exiting
07:14:32 (6744): No heartbeat from core client for 30 sec - exiting
07:14:33 (6744): No heartbeat from core client for 30 sec - exiting
07:14:35 (6744): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5960, selfPID=5960, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4744, selfPID=4744, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6848, selfPID=6848, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=560, selfPID=560, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6448, selfPID=6448, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6560, selfPID=6560, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6040, selfPID=6040, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=340, selfPID=340, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6516, selfPID=6516, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5348, selfPID=5348, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3484, selfPID=3484, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7004, selfPID=7004, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6504, selfPID=6504, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4548, selfPID=4548, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2152, selfPID=2152, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2680, selfPID=2680, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3032, selfPID=3032, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2576, selfPID=2576, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1308, selfPID=1308, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=240, selfPID=240, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7152, selfPID=7152, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2600, selfPID=2600, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6432, selfPID=6432, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7104, selfPID=7104, iMonCtr=1
12:12:11 (5228): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:12:12 (5228): No heartbeat from core client for 30 sec - exiting
12:12:13 (5228): No heartbeat from core client for 30 sec - exiting
12:12:14 (5228): No heartbeat from core client for 30 sec - exiting
12:12:15 (5228): No heartbeat from core client for 30 sec - exiting
12:12:16 (5228): No heartbeat from core client for 30 sec - exiting
12:12:17 (5228): No heartbeat from core client for 30 sec - exiting
12:12:18 (5228): No heartbeat from core client for 30 sec - exiting
12:12:19 (5228): No heartbeat from core client for 30 sec - exiting
12:12:20 (5228): No heartbeat from core client for 30 sec - exiting
12:12:21 (5228): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5508, selfPID=5508, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5268, selfPID=5268, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3396, selfPID=3396, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3156, selfPID=3156, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5936, selfPID=5936, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5108, selfPID=5108, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2700, selfPID=2700, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5840, selfPID=5840, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6728, selfPID=6728, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6176, selfPID=6176, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6848, selfPID=6848, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=676, selfPID=676, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6908, selfPID=6908, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4720, selfPID=4720, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4284, selfPID=4284, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3716, selfPID=3716, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3100, selfPID=3100, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5644, selfPID=5644, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5772, selfPID=5772, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1220, selfPID=1220, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2408, selfPID=2408, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3928, selfPID=3928, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4232, selfPID=4232, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5576, selfPID=5576, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4068, selfPID=4068, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5544, selfPID=5544, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6076, selfPID=6076, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5508, selfPID=5508, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6028, selfPID=6028, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5688, selfPID=5688, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6584, selfPID=6584, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6016, selfPID=6016, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7244, selfPID=7244, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7824, selfPID=7824, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5100, selfPID=5100, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7100, selfPID=7100, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1592, selfPID=1592, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1220, selfPID=1220, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8088, selfPID=8088, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4812, selfPID=4812, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4884, selfPID=4884, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5908, selfPID=5908, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1764, selfPID=1764, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4328, selfPID=4328, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=888, selfPID=888, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5416, selfPID=5416, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=4648, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7084, selfPID=7084, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6884, selfPID=6884, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6184, selfPID=6184, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6724, selfPID=6724, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2152, selfPID=2152, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5812, selfPID=5812, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2728, selfPID=2728, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5572, selfPID=5572, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3700, selfPID=3700, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=852, selfPID=852, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1212, selfPID=1212, iMonCtr=1

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: Result too large
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5556, selfPID=5556, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=960, selfPID=960, iMonCtr=1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
10:25:51 (1612): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Oct 2010 06:51:45 1042170 11690857 famous_v0cb_999_200_006686217_3 1,282,346 633,269 0.4938
01 Oct 2010 05:28:48 1042170 11690857 famous_v0cb_999_200_006686217_3 1,272,986 628,640 0.4938
01 Oct 2010 03:59:31 1042170 11690857 famous_v0cb_999_200_006686217_3 1,263,626 624,022 0.4938
01 Oct 2010 03:17:07 1042170 11690857 famous_v0cb_999_200_006686217_3 1,254,266 619,379 0.4938
01 Oct 2010 03:16:34 1042170 11690857 famous_v0cb_999_200_006686217_3 1,244,906 614,730 0.4938
01 Oct 2010 00:00:07 1042170 11690857 famous_v0cb_999_200_006686217_3 1,235,546 610,075 0.4938
30 Sep 2010 22:41:21 1042170 11690857 famous_v0cb_999_200_006686217_3 1,226,186 605,413 0.4937
30 Sep 2010 19:12:56 1042170 11690857 famous_v0cb_999_200_006686217_3 1,216,826 600,716 0.4937
30 Sep 2010 17:50:10 1042170 11690857 famous_v0cb_999_200_006686217_3 1,207,466 596,053 0.4936
30 Sep 2010 16:25:25 1042170 11690857 famous_v0cb_999_200_006686217_3 1,198,106 591,399 0.4936
30 Sep 2010 15:04:19 1042170 11690857 famous_v0cb_999_200_006686217_3 1,188,746 586,704 0.4935
30 Sep 2010 13:40:09 1042170 11690857 famous_v0cb_999_200_006686217_3 1,179,386 582,041 0.4935
30 Sep 2010 12:15:27 1042170 11690857 famous_v0cb_999_200_006686217_3 1,170,026 577,442 0.4935
30 Sep 2010 10:51:55 1042170 11690857 famous_v0cb_999_200_006686217_3 1,160,666 572,789 0.4935
30 Sep 2010 08:30:53 1042170 11690857 famous_v0cb_999_200_006686217_3 1,151,306 568,164 0.4935
30 Sep 2010 07:12:07 1042170 11690857 famous_v0cb_999_200_006686217_3 1,141,946 563,540 0.4935
30 Sep 2010 06:08:27 1042170 11690857 famous_v0cb_999_200_006686217_3 1,132,586 558,898 0.4935
30 Sep 2010 04:40:05 1042170 11690857 famous_v0cb_999_200_006686217_3 1,123,226 554,309 0.4935
30 Sep 2010 03:45:15 1042170 11690857 famous_v0cb_999_200_006686217_3 1,113,866 549,677 0.4935
30 Sep 2010 03:45:15 1042170 11690857 famous_v0cb_999_200_006686217_3 1,104,506 545,055 0.4935


©2024 climateprediction.net