climateprediction.net home page
Task 11833110

Task 11833110

Name famous_vm9o_1899_200_006714626_2
Workunit 6917879
Created 26 Aug 2010, 17:47:33 UTC
Sent 31 Oct 2010, 12:12:20 UTC
Report deadline 30 Jan 2011, 19:39:31 UTC
Received 30 Jan 2011, 9:10:08 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status -197 (0xFFFFFF3B) ERR_ABORTED_VIA_GUI
Computer ID 1110830
Run time 31 days 21 hours 35 min 37 sec
CPU time 21 days 22 hours 5 min 50 sec
Validate state Invalid
Credit 5,373.49
Device peak FLOPS 0.93 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
 (1277): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (1277): No heartbeat from core client for 30 sec - exiting
 (1277): No heartbeat from core client for 30 sec - exiting
 (1277): No heartbeat from core client for 30 sec - exiting

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed:                                                                                                                                                                                                                                                                 tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
 (1209): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
 (2556): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
SIGSEGV: segmentation violation
Stack trace (13 frames):
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400]
[0x266400]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x81ae177]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80d7e7a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x812c1e9]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80de1cf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x807b44a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8314811]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x832dc4d]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8330bdf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x83415d2]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x3b3bd6]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11899, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
 (25383): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400]
[0xb13400]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x81ae177]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80d7e7a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x812c1e9]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80de1cf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x807b44a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8314811]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x832dc4d]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8330bdf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x83415d2]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x267bd6]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14669, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
SIGSEGV: segmentation violation
Stack trace (14 frames):
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400]
[0x97c400]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x826d27c]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80e6a35]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80dadd1]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x812c1e9]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80de1cf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x807b44a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8314811]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x832dc4d]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8330bdf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x83415d2]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x786bd6]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21063, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (14 frames):
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400]
[0x9f5400]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x81ab784]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x81ae4f8]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80d838f]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x812c1e9]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80de1cf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x807b44a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8314811]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x832dc4d]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8330bdf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x83415d2]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x5e7bd6]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7325, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
 (12931): called boinc_finish
CPDN Monitor - Quit request from BOINC...
 (1597): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2409): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2413): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2417): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2421): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2425): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2429): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2433): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2437): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2445): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2449): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2453): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2458): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2462): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2466): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2470): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2474): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2478): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2482): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2486): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2490): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2494): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2498): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2502): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2506): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2510): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2514): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2518): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2522): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2526): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2530): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2534): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2538): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2543): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2547): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2551): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2555): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2559): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2563): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2567): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2571): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2575): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2579): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2583): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2587): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2591): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2595): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2599): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2603): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2607): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2615): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2620): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2624): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2628): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2632): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2639): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2643): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2647): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2651): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2655): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2659): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2663): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2667): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2671): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2675): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2679): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (2683): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
Stack trace (14 frames):
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400]
[0x62e400]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x826d244]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80e6a35]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80dadd1]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x812c1e9]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x80de1cf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x807b44a]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8314811]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x832dc4d]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x8330bdf]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu[0x83415d2]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x7c1bd6]
/var/lib/boinc-client/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1565, iMonCtr=1
Model crash detected, will try to restart...
 (1565): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (27932): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (27932): No heartbeat from core client for 30 sec - exiting
 (27932): No heartbeat from core client for 30 sec - exiting

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
 (28803): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14906, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
CPDN Monitor - Abort request from BOINC...
 (14906): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
30 Jan 2011 05:45:06 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,628,666 1,884,228 1.1569
30 Jan 2011 01:48:50 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,619,306 1,873,200 1.1568
29 Jan 2011 22:02:45 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,609,946 1,862,375 1.1568
29 Jan 2011 18:36:44 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,600,586 1,851,521 1.1568
29 Jan 2011 12:27:24 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,591,226 1,840,514 1.1567
29 Jan 2011 12:27:24 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,581,866 1,829,109 1.1563
29 Jan 2011 00:43:45 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,572,506 1,817,552 1.1558
28 Jan 2011 20:01:10 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,563,146 1,806,078 1.1554
28 Jan 2011 15:15:54 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,553,786 1,794,372 1.1548
28 Jan 2011 11:48:35 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,544,426 1,783,388 1.1547
28 Jan 2011 07:53:31 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,535,066 1,772,381 1.1546
28 Jan 2011 03:54:39 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,525,706 1,761,369 1.1545
28 Jan 2011 00:05:56 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,516,346 1,750,372 1.1543
27 Jan 2011 20:13:06 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,506,986 1,739,367 1.1542
27 Jan 2011 16:23:51 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,497,626 1,728,355 1.1541
27 Jan 2011 12:27:45 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,488,266 1,717,365 1.1539
27 Jan 2011 11:07:27 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,478,906 1,706,364 1.1538
27 Jan 2011 11:07:40 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,469,546 1,695,346 1.1537
27 Jan 2011 11:07:41 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,460,186 1,684,551 1.1537
26 Jan 2011 22:15:20 1110830 11833110 famous_vm9o_1899_200_006714626_2 1,450,826 1,673,753 1.1537


©2024 climateprediction.net