climateprediction.net home page
Task 20106508

Task 20106508

Name wah2_eas50_a0pv_200112_13_486_010814478_0
Workunit 10814478
Created 14 Dec 2016, 15:10:39 UTC
Sent 14 Dec 2016, 15:38:29 UTC
Report deadline 26 Nov 2017, 20:58:29 UTC
Received 22 Dec 2016, 11:02:18 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1329364
Run time 3 days 23 hours 18 min 57 sec
CPU time 3 days 19 hours 9 min 58 sec
Validate state Invalid
Credit 9,138.96
Device peak FLOPS 3.98 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 253.04 MB
Peak swap size 216.46 MB
Peak disk usage 112.36 MB
Stderr
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
finish file present too long
</message>
<stderr_txt>
15:39:24 (1184): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
15:39:25 (17912): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
11:53:58 (15820): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
11:53:59 (8524): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15820, selfPID=15820, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15820, selfPID=8524, iMonCtr=1
10:03:16 (10420): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
10:03:17 (13872): start_timer_thread(): CreateThread() failed, errno 0
11:28:00 (19172): start_timer_thread(): CreateThread() failed, errno 0
11:28:02 (12620): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19172, selfPID=19172, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19172, selfPID=12620, iMonCtr=1
06:37:22 (19504): start_timer_thread(): CreateThread() failed, errno 0
06:37:24 (12264): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
09:06:34 (11952): start_timer_thread(): CreateThread() failed, errno 0
09:06:36 (11536): start_timer_thread(): CreateThread() failed, errno 0
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11952, selfPID=11536, iMonCtr=1
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11952, selfPID=11952, iMonCtr=1
11:57:08 (10020): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10020, selfPID=10020, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10020, selfPID=14428, iMonCtr=1
11:31:17 (12280): start_timer_thread(): CreateThread() failed, errno 0
11:31:18 (4108): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12280, selfPID=12280, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12280, selfPID=4108, iMonCtr=1
15:35:36 (16112): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16112, selfPID=16112, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16112, selfPID=16216, iMonCtr=1
09:28:35 (19216): start_timer_thread(): CreateThread() failed, errno 0
09:28:37 (9436): start_timer_thread(): CreateThread() failed, errno 0
10:37:59 (6668): start_timer_thread(): CreateThread() failed, errno 0
10:38:00 (9000): start_timer_thread(): CreateThread() failed, errno 0
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9000, selfPID=18756, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
10:41:12 (18756): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0287E12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 12/22/16 10:42:43
Install Directory : 
Data Directory    : D:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\projects\climateprediction.net;D:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 D:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077cd0000 0000000000183000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.14393.479) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.14393.206 (rs1_release.160915-0644)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.206

ModLoad: 0000000077bf0000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.14393.206 (rs1_release.160915-0644)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.206

ModLoad: 0000000077330000 00000000001a1000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.14393.479) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.14393.206 (rs1_release.160915-0644)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.206

ModLoad: 000000005eff0000 0000000000067000 C:\Program Files (x86)\Norton Security with Backup\NortonData\22.6.0.142\Definitions\BASHDefs\20161215.001\UMEngx86.dll (11.0.0.357) (-exported- Symbols Loaded)
    Linked PDB Filename   : C:\bld_area\BehaviorBlocking\BB11.0\Src\Trunk\Bin\Win32\Release\UMEngx86.pdb
    File Version          : 11.0.0.357
    Company Name          : Symantec Corporation
    Product Name          : BASH
    Product Version       : 11.0.0.357

ModLoad: 0000000076e40000 000000000015f000 C:\WINDOWS\System32\USER32.dll (6.2.14393.576) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 00000000769a0000 0000000000015000 C:\WINDOWS\System32\win32u.dll (6.2.14393.51) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.14393.51 (rs1_release_inmarket.160801-1836)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.51

ModLoad: 0000000077300000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.14393.206) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.14393.206 (rs1_release.160915-0644)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.206

ModLoad: 0000000076bf0000 000000000015b000 C:\WINDOWS\System32\gdi32full.dll (6.2.14393.576) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.14393.576 (rs1_release_inmarket.161208-2252)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.576

ModLoad: 0000000076d50000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 0000000076b30000 00000000000be000 C:\WINDOWS\System32\msvcrt.dll (7.0.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.14393.0

ModLoad: 0000000076fb0000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 00000000770c0000 00000000000c1000 C:\WINDOWS\System32\RPCRT4.dll (6.2.14393.82) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 0000000074790000 000000000001e000 C:\WINDOWS\System32\SspiCli.dll (6.2.14393.576) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.14393.576 (rs1_release_inmarket.161208-2252)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.576

ModLoad: 0000000074780000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 0000000077830000 000000000005a000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 00000000772d0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 00000000730a0000 0000000000145000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.14321.1024) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.14321.1024 (rs1_release.160715-1616)
    Company Name          : Microsoft
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14321.1024

ModLoad: 0000000076780000 00000000000e0000 C:\WINDOWS\System32\ucrtbase.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0

ModLoad: 0000000072fd0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.14393.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.14393.0 (rs1_release.160715-1616)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.14393.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 1660, Write: 0, Other 15664

- I/O Transfers Counters -
Read: 0, Write: 1932, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 86872, QuotaPeakPagedPoolUsage: 87000
QuotaNonPagedPoolUsage: 9976, QuotaPeakNonPagedPoolUsage: 10400

- Virtual Memory Usage -
VirtualSize: 55033856, PeakVirtualSize: 55033856

- Pagefile Usage -
PagefileUsage: 3510272, PeakPagefileUsage: 3510272

- Working Set Size -
WorkingSetSize: 6176768, PeakWorkingSetSize: 6176768, PageFaultCount: 1824

*** Dump of thread ID 9488 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 3593750.000000, User Time: 3750000.000000, Wait Time: 3380.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0287E12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc88 edx=0287e12c esi=0019da40 edi=0019dcb8
eip=0048ee46 esp=0019d9d4 ebp=0019dc64
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc64 0048161c 0019dc88 004b63b8 00000000 0019dcb8 wah2_8.24_windows_intelx86!+0x0 
0019dca8 0044fa52 0287e12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dce0 00485ca1 c0000005 0019dd14 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 77c062c4 002b9000 77c062a0 b023c70a 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 77d30fd9 002b9000 0c1a6823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 
0019ffec 00000000 00485cce 002b9000 00000000 78746341 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 

*** Dump of thread ID 9616 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2009.000000

- Registers -
eax=00000000 ebx=00803478 ecx=00000000 edx=00000000 esi=00803a70 edi=00803c30
eip=77d401cc esp=007bfddc ebp=007bff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
007bff80 77c062c4 00803478 77c062a0 b041c70a 007bffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
007bff94 77d30fd9 00803478 0c786823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
007bffdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 
007bffec 00000000 77d067c0 00803478 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 

*** Dump of thread ID 9620 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2009.000000

- Registers -
eax=00000000 ebx=00803478 ecx=00000000 edx=00000000 esi=00808600 edi=008087c0
eip=77d401cc esp=009ffddc ebp=009fff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009fff80 77c062c4 00803478 77c062a0 b0a5c70a 009fffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
009fff94 77d30fd9 00803478 0c9c6823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009fffdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 
009fffec 00000000 77d067c0 00803478 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 

*** Dump of thread ID 9624 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2009.000000

- Registers -
eax=00000000 ebx=00803478 ecx=00000000 edx=00000000 esi=00808860 edi=00808a20
eip=77d401cc esp=00b3fddc ebp=00b3ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00b3ff80 77c062c4 00803478 77c062a0 b089c70a 00b3ffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00b3ff94 77d30fd9 00803478 0cb06823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b3ffdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 
00b3ffec 00000000 77d067c0 00803478 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 

*** Dump of thread ID 9628 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3377.000000

- Registers -
eax=00000000 ebx=001f0000 ecx=00000000 edx=00000000 esi=001f0064 edi=001f0064
eip=001d039a esp=00c7ff50 ebp=00c7ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000216

- Callstack -
ChildEBP RetAddr  Args to Child
00c7ff80 77c062c4 001f0000 77c062a0 b0fdc70a 00c7ffdc !+0x0 SymFromAddr(): GetLastError = '126'  SymGetModuleInfo(): GetLastError = '126' Address = '001d039a'
00c7ff94 77d30fd9 001f0000 0cc46823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 SymFromAddr(): GetLastError = '126'  SymGetModuleInfo(): GetLastError = '126' Address = '77c062c4'
00c7ffdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 SymFromAddr(): GetLastError = '126'  SymGetModuleInfo(): GetLastError = '126' Address = '77d30fd9'
00c7ffec 00000000 001f0064 001f0000 00000000 03000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 SymFromAddr(): GetLastError = '126'  SymGetModuleInfo(): GetLastError = '126' Address = '77d30fa4'

*** Dump of thread ID 9640 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3377.000000

- Registers -
eax=00000000 ebx=027afd1c ecx=00000000 edx=00000000 esi=00000000 edi=027afd1c
eip=77d3e91c esp=027afcdc ebp=027afd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
027afd40 77402bcf 00000064 00000000 027aff80 00478b44 ntdll!NtDelayExecution+0x0 
027afd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
027aff80 77c062c4 00000000 77c062a0 b240c70a 027affdc wah2_8.24_windows_intelx86!+0x0 
027aff94 77d30fd9 00000000 0e796823 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
027affdc 77d30fa4 ffffffff 77d52efc 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 
027affec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
21 Dec 2016 18:59:10 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 138,539 310,408 2.2406
20 Dec 2016 22:31:29 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 127,019 285,963 2.2513
20 Dec 2016 13:44:02 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 115,499 259,580 2.2475
19 Dec 2016 16:33:46 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 103,979 233,227 2.2430
18 Dec 2016 22:23:23 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 92,459 206,849 2.2372
18 Dec 2016 15:04:07 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 80,939 180,926 2.2353
18 Dec 2016 08:20:10 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 69,419 155,094 2.2342
17 Dec 2016 17:45:18 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 57,899 128,984 2.2277
16 Dec 2016 22:07:17 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 46,379 102,451 2.2090
16 Dec 2016 14:07:48 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 34,859 76,427 2.1925
15 Dec 2016 20:08:05 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 23,339 50,961 2.1835
15 Dec 2016 12:57:41 1329364 20106508 wah2_eas50_a0pv_200112_13_486_010814478_0 11,819 25,737 2.1776


©2024 climateprediction.net