climateprediction.net home page
Task 12514617

Task 12514617

Name famous_xdtq_1999_200_007079749_2
Workunit 7283049
Created 23 Jan 2011, 10:52:55 UTC
Sent 23 Jan 2011, 10:52:59 UTC
Report deadline 24 Apr 2011, 18:20:10 UTC
Received 24 Jan 2011, 1:39:12 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1118595
Run time 13 hours 54 min 56 sec
CPU time 12 hours 30 min 11 sec
Validate state Invalid
Credit 586.84
Device peak FLOPS 2.20 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 10:53:21 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd   11  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd48c edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4c8 esp: 0xbfffd48c
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=63856, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 16:34:37 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd   11  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd48c edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4c8 esp: 0xbfffd48c
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69329, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 16:34:39 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd   12  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69329, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 16:40:40 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd   11  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd48c edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4c8 esp: 0xbfffd48c
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69452, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 17:01:08 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd   12  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69817, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 17:02:49 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd   11  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd48c edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4c8 esp: 0xbfffd48c
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69865, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Sun Jan 23 17:05:19 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd   12  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69931, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Mon Jan 24 01:34:58 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x96d4b46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd   12  0x00000003  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc3fc edx: 0x96ce50fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc438 esp: 0xbfffc3fc
   ss: 0x0000001f efl: 0x00000206 eip: 0x96ce50fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_xdtq_1999_200_007079749/lib/libirc.dylib
0x958fe000 - 0x95901fff /usr/lib/system/libmathCommon.A.dylib
0x96ce4000 - 0x96e8bfff /usr/lib/libSystem.B.dylib
0x99268000 - 0x99276fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=76873, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (76873): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
24 Jan 2011 01:32:38 1118595 12514617 famous_xdtq_1999_200_007079749_2 177,866 44,628 0.2509
24 Jan 2011 00:51:09 1118595 12514617 famous_xdtq_1999_200_007079749_2 168,506 42,287 0.2510
24 Jan 2011 00:09:47 1118595 12514617 famous_xdtq_1999_200_007079749_2 159,146 39,945 0.2510
23 Jan 2011 23:28:13 1118595 12514617 famous_xdtq_1999_200_007079749_2 149,786 37,601 0.2510
23 Jan 2011 22:46:06 1118595 12514617 famous_xdtq_1999_200_007079749_2 140,426 35,258 0.2511
23 Jan 2011 22:08:17 1118595 12514617 famous_xdtq_1999_200_007079749_2 131,066 32,915 0.2511
23 Jan 2011 21:22:09 1118595 12514617 famous_xdtq_1999_200_007079749_2 121,706 30,569 0.2512
23 Jan 2011 20:40:50 1118595 12514617 famous_xdtq_1999_200_007079749_2 112,346 28,228 0.2513
23 Jan 2011 19:53:48 1118595 12514617 famous_xdtq_1999_200_007079749_2 102,986 25,849 0.2510
23 Jan 2011 19:07:44 1118595 12514617 famous_xdtq_1999_200_007079749_2 93,626 23,470 0.2507
23 Jan 2011 18:21:40 1118595 12514617 famous_xdtq_1999_200_007079749_2 84,266 21,107 0.2505
23 Jan 2011 18:02:27 1118595 12514617 famous_xdtq_1999_200_007079749_2 74,906 18,769 0.2506
23 Jan 2011 16:26:48 1118595 12514617 famous_xdtq_1999_200_007079749_2 65,546 16,404 0.2503
23 Jan 2011 15:23:25 1118595 12514617 famous_xdtq_1999_200_007079749_2 56,186 14,150 0.2518
23 Jan 2011 14:40:03 1118595 12514617 famous_xdtq_1999_200_007079749_2 46,826 11,812 0.2523
23 Jan 2011 14:03:00 1118595 12514617 famous_xdtq_1999_200_007079749_2 37,466 9,470 0.2528
23 Jan 2011 13:32:50 1118595 12514617 famous_xdtq_1999_200_007079749_2 28,106 7,119 0.2533
23 Jan 2011 12:27:35 1118595 12514617 famous_xdtq_1999_200_007079749_2 18,746 4,739 0.2528
23 Jan 2011 11:42:23 1118595 12514617 famous_xdtq_1999_200_007079749_2 9,386 2,358 0.2512


©2024 climateprediction.net