climateprediction.net home page
Task 11899215

Task 11899215

Name hadsm3dhet2_u2fc_006725879_9
Workunit 6929222
Created 17 Sep 2010, 8:08:41 UTC
Sent 20 Sep 2010, 23:35:51 UTC
Report deadline 3 Sep 2011, 4:55:51 UTC
Received 9 Oct 2010, 23:29:52 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 963734
Run time 11 days 3 hours 32 min 12 sec
CPU time 10 days 17 hours 20 min 51 sec
Validate state Invalid
Credit 6,351.58
Device peak FLOPS 2.63 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.07
windows_intelx86
Stderr
<core_client_version>6.6.28</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
MainError:	11:39:35 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=4000, selfPID=4000, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
MainError:	10:35:15 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No Process Handle
CPDN process is not running, exiting, bRetVal = 1, checkPID=6836, selfPID=6836, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: (null)


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7D4E237E

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.3.8


Dump Timestamp    : 10/08/10 23:24:55
Install Directory : 
Data Directory    : C:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( C:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadsm3dhet2_u2fc_006725879;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net;srv*C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\symbols*http://msdl.microsoft.com/download/symbols;srv*C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/symstore


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2, Write: 0, Other 186

- I/O Transfers Counters -
Read: 0, Write: 55, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 90320, QuotaPeakPagedPoolUsage: 90320
QuotaNonPagedPoolUsage: 5216, QuotaPeakNonPagedPoolUsage: 5216

- Virtual Memory Usage -
VirtualSize: 241008640, PeakVirtualSize: 241008640

- Pagefile Usage -
PagefileUsage: 7839744, PeakPagefileUsage: 9437184

- Working Set Size -
WorkingSetSize: 3366912, PeakWorkingSetSize: 3366912, PageFaultCount: 837

*** Dump of thread ID 2860 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 123369216.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7D4E237E

- Registers -
eax=0487fccc ebx=00000001 ecx=00000000 edx=00000000 esi=0487fd54 edi=fffffffe
eip=7d4e237e esp=0487fcc8 ebp=0487fd1c
cs=120023  ss=12002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0487fd1c 005c1c17 e06d7363 00000001 00000003 0487fd48 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e237e'
0487fd54 005c1bce 0487fd64 0069ecb8 0065bb7c 0065b39c !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005c1c17'
0487fd70 004096c6 00200000 0040bb2b 060a0000 060a3a06 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005c1bce'
0487ffc0 7d4e7d42 00000000 00000000 7efde000 e06d7363 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004096c6'
0487fff0 00000000 005c570d 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7d42'

*** Dump of thread ID 4664 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 123369216.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=0db5ff74
eip=7d61ccc6 esp=0db5ff38 ebp=0db5ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0db5ff9c 7d4d14ef 00000064 00000000 0db5ffb8 004213b4 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
0db5ffac 004213b4 00000064 0db5ffec 7d4dfe37 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
0db5ffb8 7d4dfe37 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004213b4'
0db5ffec 00000000 004213a0 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfe37'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Oct 2010 23:33:33 963734 11899215 hadsm3dhet2_u2fc_006725879_9 172,832 917,453 1.3271
09 Oct 2010 23:33:33 963734 11899215 hadsm3dhet2_u2fc_006725879_9 162,030 903,215 1.3272
09 Oct 2010 23:33:33 963734 11899215 hadsm3dhet2_u2fc_006725879_9 151,228 888,980 1.3274
06 Oct 2010 04:22:19 963734 11899215 hadsm3dhet2_u2fc_006725879_9 140,426 875,425 1.3286
05 Oct 2010 18:24:25 963734 11899215 hadsm3dhet2_u2fc_006725879_9 129,624 861,801 1.3297
05 Oct 2010 13:13:07 963734 11899215 hadsm3dhet2_u2fc_006725879_9 118,822 848,147 1.3308
05 Oct 2010 08:34:27 963734 11899215 hadsm3dhet2_u2fc_006725879_9 108,020 833,102 1.3297
05 Oct 2010 03:48:10 963734 11899215 hadsm3dhet2_u2fc_006725879_9 97,218 817,833 1.3283
04 Oct 2010 23:04:34 963734 11899215 hadsm3dhet2_u2fc_006725879_9 86,416 802,585 1.3268
04 Oct 2010 18:19:14 963734 11899215 hadsm3dhet2_u2fc_006725879_9 75,614 787,374 1.3253
04 Oct 2010 13:53:04 963734 11899215 hadsm3dhet2_u2fc_006725879_9 64,812 773,181 1.3255
04 Oct 2010 03:11:03 963734 11899215 hadsm3dhet2_u2fc_006725879_9 54,010 759,391 1.3264
03 Oct 2010 21:50:21 963734 11899215 hadsm3dhet2_u2fc_006725879_9 43,208 745,722 1.3276
03 Oct 2010 12:27:02 963734 11899215 hadsm3dhet2_u2fc_006725879_9 32,406 732,113 1.3289
02 Oct 2010 19:45:41 963734 11899215 hadsm3dhet2_u2fc_006725879_9 21,604 717,278 1.3280
02 Oct 2010 15:11:26 963734 11899215 hadsm3dhet2_u2fc_006725879_9 10,802 702,716 1.3276
02 Oct 2010 10:36:04 963734 11899215 hadsm3dhet2_u2fc_006725879_9 259,248 688,136 1.3272
02 Oct 2010 03:07:36 963734 11899215 hadsm3dhet2_u2fc_006725879_9 248,446 673,705 1.3270
01 Oct 2010 11:17:46 963734 11899215 hadsm3dhet2_u2fc_006725879_9 237,644 658,892 1.3260
01 Oct 2010 06:29:16 963734 11899215 hadsm3dhet2_u2fc_006725879_9 226,842 643,630 1.3241


©2024 climateprediction.net