 |
|
| Name | famous_wua6_1999_200_007113749_1 |
| Workunit | 7313204 |
| Created | 26 Feb 2011, 17:49:10 UTC |
| Sent | 26 Feb 2011, 17:49:19 UTC |
| Report deadline | 29 May 2011, 1:16:30 UTC |
| Received | 1 Apr 2011, 11:34:13 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 22 (0x00000016) Unknown error code |
| Computer ID | 1107292 |
| Run time | 22 days 8 hours 9 min 54 sec |
| CPU time | 16 days 20 hours 26 min 41 sec |
| Validate state | Invalid |
| Credit | 5,342.61 |
| Device peak FLOPS | 1.82 GFLOPS |
| Application version | UK Met Office FAMOUS v6.11 i686-pc-linux-gnu |
| Stderr | <core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Suspended CPDN Monitor - Suspend request from BOINC...
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1
BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
CPDN Monitor - Quit request from BOINC...
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Signal 15 received, exiting...
Signal 15 received, exiting...
(14078): called boinc_finish
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Signal 15 received, exiting...
(1351): called boinc_finish
SIGSEGV: segmentation violation
Stack trace (7 frames):
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c]
[0xffffe400]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6]
/lib/libc.so.6(__libc_start_main+0xfe)[0xf74ddc0e]
../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449]
Exiting...
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Suspended CPDN Monitor - Suspend request from BOINC...
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy
Sorry, too many model crashes! :-(
(1301): called boinc_finish
</stderr_txt>
]]>
|
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 01 Apr 2011 09:15:14 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,619,306 | 1,451,551 | 0.8964 |
| 01 Apr 2011 06:58:42 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,609,946 | 1,443,575 | 0.8967 |
| 01 Apr 2011 04:44:15 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,600,586 | 1,435,597 | 0.8969 |
| 01 Apr 2011 02:15:46 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,591,226 | 1,427,623 | 0.8972 |
| 31 Mar 2011 23:54:12 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,581,866 | 1,419,647 | 0.8975 |
| 31 Mar 2011 21:33:47 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,572,506 | 1,411,486 | 0.8976 |
| 31 Mar 2011 19:13:24 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,563,146 | 1,403,466 | 0.8978 |
| 31 Mar 2011 17:20:16 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,553,786 | 1,395,508 | 0.8981 |
| 31 Mar 2011 14:24:11 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,544,426 | 1,387,183 | 0.8982 |
| 31 Mar 2011 12:01:48 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,535,066 | 1,379,218 | 0.8985 |
| 31 Mar 2011 09:43:49 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,525,706 | 1,371,242 | 0.8988 |
| 31 Mar 2011 07:03:22 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,516,346 | 1,362,735 | 0.8987 |
| 31 Mar 2011 04:46:06 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,506,986 | 1,354,787 | 0.8990 |
| 31 Mar 2011 02:27:27 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,497,626 | 1,346,845 | 0.8993 |
| 31 Mar 2011 00:02:58 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,488,266 | 1,338,584 | 0.8994 |
| 30 Mar 2011 21:40:57 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,478,906 | 1,330,640 | 0.8997 |
| 30 Mar 2011 19:20:27 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,469,546 | 1,322,695 | 0.9001 |
| 30 Mar 2011 17:00:48 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,460,186 | 1,314,744 | 0.9004 |
| 30 Mar 2011 14:42:08 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,450,826 | 1,306,800 | 0.9007 |
| 30 Mar 2011 12:21:11 | 1107292 | 12638902 | famous_wua6_1999_200_007113749_1 | 1,441,466 | 1,298,841 | 0.9011 |
©2024 climateprediction.net