climateprediction.net home page
Task 12505660

Task 12505660

Name famous_x800_599_200_007072199_1
Workunit 7275499
Created 19 Jan 2011, 13:24:11 UTC
Sent 19 Jan 2011, 13:30:07 UTC
Report deadline 20 Apr 2011, 20:57:18 UTC
Received 23 Jan 2011, 12:55:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1115102
Run time 15 hours 59 min 56 sec
CPU time 14 hours 40 min 44 sec
Validate state Invalid
Credit 833.89
Device peak FLOPS 2.79 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Thu Jan 20 20:49:19 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2565, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Fri Jan 21 07:57:36 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2937, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Fri Jan 21 18:32:20 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3289, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Fri Jan 21 18:33:40 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3313, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Fri Jan 21 20:01:15 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3663, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Sat Jan 22 17:50:54 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=664, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.3 build 10D2125
Sat Jan 22 21:37:28 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x9934f1fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd48c edx: 0x992e90fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4c8 esp: 0xbfffd48c
   ss: 0x0000001f efl: 0x00000206 eip: 0x992e90fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_x800_599_200_007072199/lib/libirc.dylib
0x9440b000 - 0x94419fff /usr/lib/libz.1.dylib
0x98ab9000 - 0x98abcfff /usr/lib/system/libmathCommon.A.dylib
0x992e8000 - 0x9948efff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=930, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (172): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
22 Jan 2011 19:30:24 1115102 12505660 famous_x800_599_200_007072199_1 252,746 51,452 0.2036
22 Jan 2011 17:57:15 1115102 12505660 famous_x800_599_200_007072199_1 243,386 49,548 0.2036
22 Jan 2011 01:40:38 1115102 12505660 famous_x800_599_200_007072199_1 234,026 47,656 0.2036
21 Jan 2011 19:58:35 1115102 12505660 famous_x800_599_200_007072199_1 224,666 45,747 0.2036
21 Jan 2011 17:00:09 1115102 12505660 famous_x800_599_200_007072199_1 215,306 43,831 0.2036
21 Jan 2011 16:28:50 1115102 12505660 famous_x800_599_200_007072199_1 205,946 41,929 0.2036
21 Jan 2011 15:52:37 1115102 12505660 famous_x800_599_200_007072199_1 196,586 40,025 0.2036
21 Jan 2011 14:43:58 1115102 12505660 famous_x800_599_200_007072199_1 187,226 38,113 0.2036
21 Jan 2011 13:43:21 1115102 12505660 famous_x800_599_200_007072199_1 177,866 36,204 0.2035
21 Jan 2011 13:13:02 1115102 12505660 famous_x800_599_200_007072199_1 168,506 34,315 0.2036
21 Jan 2011 12:36:06 1115102 12505660 famous_x800_599_200_007072199_1 159,146 32,411 0.2037
21 Jan 2011 12:27:59 1115102 12505660 famous_x800_599_200_007072199_1 149,786 30,475 0.2035
21 Jan 2011 12:27:59 1115102 12505660 famous_x800_599_200_007072199_1 140,426 28,567 0.2034
20 Jan 2011 21:01:30 1115102 12505660 famous_x800_599_200_007072199_1 131,066 26,657 0.2034
20 Jan 2011 20:23:10 1115102 12505660 famous_x800_599_200_007072199_1 121,706 24,739 0.2033
20 Jan 2011 19:52:15 1115102 12505660 famous_x800_599_200_007072199_1 112,346 22,838 0.2033
20 Jan 2011 19:36:52 1115102 12505660 famous_x800_599_200_007072199_1 102,986 20,934 0.2033
20 Jan 2011 19:36:52 1115102 12505660 famous_x800_599_200_007072199_1 93,626 19,030 0.2033
20 Jan 2011 18:30:16 1115102 12505660 famous_x800_599_200_007072199_1 84,266 17,124 0.2032
20 Jan 2011 18:23:51 1115102 12505660 famous_x800_599_200_007072199_1 74,906 15,220 0.2032


©2024 climateprediction.net