climateprediction.net home page
Task 20646312

Task 20646312

Name wah2_wus25_t522_199909_25_622_011153798_1
Workunit 11153798
Created 10 Aug 2017, 2:32:03 UTC
Sent 30 Aug 2017, 9:51:00 UTC
Report deadline 12 Aug 2018, 15:11:00 UTC
Received 14 Sep 2017, 22:37:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1339864
Run time 11 days 9 hours 59 min 53 sec
CPU time 9 days 20 hours 45 min 49 sec
Validate state Invalid
Credit 10,658.83
Device peak FLOPS 3.65 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 225.81 MB
Peak swap size 165.12 MB
Peak disk usage 0.02 MB
Stderr
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
finish file present too long
</message>
<stderr_txt>
05:54:30 (15748): start_timer_thread(): CreateThread() failed, errno 0
05:54:32 (8768): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
14:06:33 (2344): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
20:30:39 (9488): start_timer_thread(): CreateThread() failed, errno 0
20:30:41 (19000): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
10:58:13 (14240): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
10:58:15 (19220): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
11:57:25 (14280): start_timer_thread(): CreateThread() failed, errno 0
11:57:27 (10728): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:10:55 (19532): start_timer_thread(): CreateThread() failed, errno 0
13:10:57 (19536): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:36:21 (14984): start_timer_thread(): CreateThread() failed, errno 0
15:36:22 (13144): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
16:28:43 (20584): start_timer_thread(): CreateThread() failed, errno 0
16:28:45 (4564): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:38:39 (14036): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:49:57 (21024): start_timer_thread(): CreateThread() failed, errno 0
18:49:59 (18108): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
22:26:21 (20856): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
23:00:39 (14452): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
02:27:42 (2124): start_timer_thread(): CreateThread() failed, errno 0
02:27:47 (244): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
03:27:02 (8472): start_timer_thread(): CreateThread() failed, errno 0
03:27:04 (15616): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:48:32 (20052): start_timer_thread(): CreateThread() failed, errno 0
13:48:33 (14604): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:30:03 (2216): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:56:34 (2552): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
15:56:36 (20736): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
00:35:29 (10904): start_timer_thread(): CreateThread() failed, errno 0
00:35:31 (21132): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
07:28:11 (9860): start_timer_thread(): CreateThread() failed, errno 0
07:28:13 (13004): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
20:21:55 (10520): start_timer_thread(): CreateThread() failed, errno 0
20:21:56 (2848): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
00:27:09 (16124): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:31:06 (18260): start_timer_thread(): CreateThread() failed, errno 0
15:31:07 (5500): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
14:46:24 (2124): start_timer_thread(): CreateThread() failed, errno 0
14:46:25 (14140): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
16:54:57 (14696): start_timer_thread(): CreateThread() failed, errno 0
16:55:00 (10412): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
20:48:32 (14436): start_timer_thread(): CreateThread() failed, errno 0
20:48:34 (10372): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
10:15:56 (11852): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
11:03:27 (15792): start_timer_thread(): CreateThread() failed, errno 0
11:03:29 (9104): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:27:30 (14296): start_timer_thread(): CreateThread() failed, errno 0
12:27:31 (2452): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:54:12 (17568): start_timer_thread(): CreateThread() failed, errno 0
13:54:13 (16924): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:41:50 (13720): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:44:51 (12128): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15136, selfPID=17284, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
17:27:28 (17284): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0272712C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 09/14/17 17:53:40
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000778b0000 000000000018e000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation Microsoft&#174; Windows&#174;
    Product Version       : 10.0.15063.608

ModLoad: 0000000075f10000 00000000000d0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.15063.296) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation Microsoft&#174; Windows&#174;
    Product Version       : 10.0.15063.608

ModLoad: 0000000076e40000 00000000001c2000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation Microsoft&#174; Windows&#174;
    Product Version       : 10.0.15063.608

ModLoad: 00000000744b0000 000000000013c000 C:\WINDOWS\System32\USER32.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation MicrosoftMD WindowsMD
    Product Version       : 10.0.15063.0

ModLoad: 0000000074470000 0000000000016000 C:\WINDOWS\System32\win32u.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076500000 0000000000021000 C:\WINDOWS\System32\GDI32.dll (6.2.15063.483) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.15063.483 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.483

ModLoad: 0000000076160000 0000000000158000 C:\WINDOWS\System32\gdi32full.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000077600000 0000000000079000 C:\WINDOWS\System32\msvcp_win.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000077730000 0000000000118000 C:\WINDOWS\System32\ucrtbase.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.15063.447 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.447

ModLoad: 0000000074680000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation Microsoft&#174; Windows&#174;
    Product Version       : 10.0.15063.0

ModLoad: 00000000767b0000 00000000000bd000 C:\WINDOWS\System32\msvcrt.dll (7.0.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.15063.0

ModLoad: 00000000776e0000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000743a0000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Syst&#232;me d&#146;exploitation Microsoft&#174; Windows&#174;
    Product Version       : 10.0.15063.0

ModLoad: 0000000074380000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000074370000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000076650000 0000000000057000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.15063.138) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.15063.138 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.138

ModLoad: 00000000765c0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000072a50000 000000000015d000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.15063.447 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.447

ModLoad: 0000000073e60000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 656, Write: 0, Other 5502

- I/O Transfers Counters -
Read: 0, Write: 1946, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 90480, QuotaPeakPagedPoolUsage: 90640
QuotaNonPagedPoolUsage: 8352, QuotaPeakNonPagedPoolUsage: 8776

- Virtual Memory Usage -
VirtualSize: 55390208, PeakVirtualSize: 55390208

- Pagefile Usage -
PagefileUsage: 3420160, PeakPagefileUsage: 3420160

- Working Set Size -
WorkingSetSize: 6246400, PeakWorkingSetSize: 6246400, PageFaultCount: 1851

*** Dump of thread ID 12464 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 1406250.000000, User Time: 1718750.000000, Wait Time: 73024.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0272712C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019d8e8 edx=0272712c esi=0019d6a0 edi=0019d918
eip=0048ee46 esp=0019d634 ebp=0019d8c4
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019d8c4 0048161c 0019d8e8 004b63b8 00000000 0019d918 wah2_8.24_windows_intelx86!+0x0 
0019d908 0044fa52 0272712c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019d928 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019d940 00485ca1 c0000005 0019d974 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 75f28744 0020a000 75f28720 35cbd390 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 7791582d 0020a000 fb4a663e 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 779157fd ffffffff 77936366 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 0020a000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 9888 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 71493.000000

- Registers -
eax=00000000 ebx=00804b68 ecx=00000000 edx=00000000 esi=00805190 edi=00805350
eip=7792396c esp=009ffd7c ebp=009fff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009fff80 75f28744 00804b68 75f28720 354dd390 009fffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
009fff94 7791582d 00804b68 fbcc663e 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009fffdc 779157fd ffffffff 77936366 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
009fffec 00000000 778ea470 00804b68 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 12736 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 71493.000000

- Registers -
eax=00000000 ebx=00804b68 ecx=00000000 edx=00000000 esi=00808b20 edi=00808ce0
eip=7792396c esp=00affd7c ebp=00afff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00afff80 75f28744 00804b68 75f28720 357dd390 00afffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
00afff94 7791582d 00804b68 fbfc663e 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00afffdc 779157fd ffffffff 77936366 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00afffec 00000000 778ea470 00804b68 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 11232 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 71493.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000000
eip=00000000 esp=00000000 ebp=00000000
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000

- Callstack -
ChildEBP RetAddr  Args to Child
(-nosymbols- PC == 0)
00000000 00000000 00000000 00000000 00000000 00000000 !RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 14492 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73019.000000

- Registers -
eax=00000000 ebx=0265fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=7792205c esp=0265fcdc ebp=0265fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0265fd40 76f3354f 00000064 00000000 0265ff80 00478b44 ntdll!NtDelayExecution+0x0 
0265fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0265ff80 75f28744 00000000 75f28720 37b7d390 0265ffdc wah2_8.24_windows_intelx86!+0x0 
0265ff94 7791582d 00000000 f936663e 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0265ffdc 779157fd ffffffff 77936366 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0265ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
13 Sep 2017 01:52:11 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 161,579 800,642 4.9551
11 Sep 2017 20:37:41 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 150,059 742,916 4.9508
11 Sep 2017 02:11:03 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 138,539 692,114 4.9958
10 Sep 2017 07:18:23 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 127,019 642,188 5.0558
09 Sep 2017 13:53:46 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 115,499 583,588 5.0528
08 Sep 2017 23:52:35 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 103,979 531,806 5.1146
07 Sep 2017 03:59:31 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 92,459 474,311 5.1300
05 Sep 2017 10:44:03 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 80,939 417,412 5.1571
04 Sep 2017 17:49:02 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 69,419 361,380 5.2058
03 Sep 2017 23:35:52 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 57,899 300,866 5.1964
03 Sep 2017 05:57:58 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 46,379 240,583 5.1873
02 Sep 2017 13:18:38 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 34,859 185,031 5.3080
01 Sep 2017 07:25:11 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 23,339 127,866 5.4786
31 Aug 2017 11:17:09 1339864 20646312 wah2_wus25_t522_199909_25_622_011153798_1 11,819 63,109 5.3396


©2024 climateprediction.net