climateprediction.net home page
Task 21329678

Task 21329678

Name wah2_pnw25_pb8j_198609_28_754_011622920_2
Workunit 11622920
Created 3 Oct 2018, 16:11:08 UTC
Sent 3 Oct 2018, 16:22:16 UTC
Report deadline 15 Sep 2019, 21:42:16 UTC
Received 11 Oct 2018, 4:22:55 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1346937
Run time 5 days 23 hours 15 min 22 sec
CPU time 5 days 16 hours 51 min 58 sec
Validate state Invalid
Credit 13,698.58
Device peak FLOPS 4.21 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 229.84 MB
Peak swap size 166.40 MB
Peak disk usage 12.76 MB
Stderr 
<core_client_version>7.12.1</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
GCM : BUFFIN: C I/O Error feof - Unit 61 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

GCM : BUFFIN: C I/O Error feof - Unit 63 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 63 - Return code = 16

GCM : BUFFIN: C I/O Error feof - Unit 64 - Return code = 16
GCM : BUFFIN: C I/O Error feof - Unit 64 - Return code = 16

CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=109908, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=108244, selfPID=110140, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
17:40:03 (110140): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x026A012C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 10/10/18 20:57:22
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077150000 0000000000190000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17134.254) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.17134.228 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.228

ModLoad: 00000000767e0000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.17134.228 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.228

ModLoad: 0000000074600000 00000000001e4000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.17134.228 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.228

ModLoad: 0000000076e70000 000000000018d000 C:\WINDOWS\System32\USER32.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000766a0000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000076ab0000 0000000000022000 C:\WINDOWS\System32\GDI32.dll (6.2.17134.285) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.17134.285 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.285

ModLoad: 0000000073a50000 0000000000164000 C:\WINDOWS\System32\gdi32full.dll (6.2.17134.345) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.17134.345 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.345

ModLoad: 00000000744c0000 000000000007d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000763f0000 000000000011e000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17134.319 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.319

ModLoad: 00000000743e0000 0000000000078000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074540000 00000000000bf000 C:\WINDOWS\System32\msvcrt.dll (7.0.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17134.1

ModLoad: 00000000741f0000 0000000000044000 C:\WINDOWS\System32\sechost.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074a60000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17134.112) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000073a10000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000073a00000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000076710000 0000000000058000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.17134.345) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.17134.345 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.345

ModLoad: 00000000741a0000 0000000000026000 C:\WINDOWS\System32\IMM32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006b730000 0000000000182000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000072720000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 776, Write: 0, Other 6542

- I/O Transfers Counters -
Read: 0, Write: 2066, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 91944, QuotaPeakPagedPoolUsage: 92104
QuotaNonPagedPoolUsage: 8760, QuotaPeakNonPagedPoolUsage: 9184

- Virtual Memory Usage -
VirtualSize: 56082432, PeakVirtualSize: 56082432

- Pagefile Usage -
PagefileUsage: 3416064, PeakPagefileUsage: 3416064

- Working Set Size -
WorkingSetSize: 5898240, PeakWorkingSetSize: 5898240, PageFaultCount: 1726

*** Dump of thread ID 15164 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 1093750.000000, User Time: 3281250.000000, Wait Time: 727282.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x026A012C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc88 edx=026a012c esi=0019da40 edi=0019dcb8
eip=0048ee46 esp=0019d9d4 ebp=0019dc64
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc64 0048161c 0019dc88 004b63b8 00000000 0019dcb8 wah2_8.24_windows_intelx86!+0x0 
0019dca8 0044fa52 026a012c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dce0 00485ca1 c0000005 0019dd14 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 767f8484 002ed000 767f8460 0b212cd5 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 771b305a 002ed000 e57a65d5 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 771b302a ffffffff 771cec8d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
0019ffec 00000000 00485cce 002ed000 00000000 78746341 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 15060 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 726437.000000

- Registers -
eax=00000000 ebx=00648468 ecx=00000000 edx=00000000 esi=006482a8 edi=006440d0
eip=771bbf1c esp=0087fdc8 ebp=0087ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0087ff80 767f8484 006440d0 767f8460 0bbf2cd5 0087ffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
0087ff94 771b305a 006440d0 e5e465d5 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0087ffdc 771b302a ffffffff 771cec8d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
0087ffec 00000000 7719c920 006440d0 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 15076 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 726437.000000

- Registers -
eax=00000000 ebx=00648db8 ecx=00000000 edx=00000000 esi=00648bf8 edi=006440d0
eip=771bbf1c esp=009bfdc8 ebp=009bff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009bff80 767f8484 006440d0 767f8460 0ba32cd5 009bffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
009bff94 771b305a 006440d0 e5f865d5 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009bffdc 771b302a ffffffff 771cec8d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
009bffec 00000000 7719c920 006440d0 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 15016 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 726437.000000

- Registers -
eax=00000000 ebx=006499c8 ecx=00000000 edx=00000000 esi=00649808 edi=006440d0
eip=771bbf1c esp=00affdc8 ebp=00afff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00afff80 767f8484 006440d0 767f8460 0b972cd5 00afffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00afff94 771b305a 006440d0 e5cc65d5 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00afffdc 771b302a ffffffff 771cec8d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
00afffec 00000000 7719c920 006440d0 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 13632 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 727278.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=771ba5ac esp=025dfcdc ebp=025dfd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
025dfd40 747070af 00000064 00000000 025dff80 00478b44 ntdll!NtDelayExecution+0x0 
025dfd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
025dff80 767f8484 00000000 767f8460 09652cd5 025dffdc wah2_8.24_windows_intelx86!+0x0 
025dff94 771b305a 00000000 e73e65d5 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
025dffdc 771b302a ffffffff 771cec8d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
025dffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Oct 2018 16:50:01 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 207,659 469,214 2.2595
10 Oct 2018 08:11:15 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 196,139 442,566 2.2564
10 Oct 2018 00:02:35 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 184,619 415,448 2.2503
09 Oct 2018 14:12:57 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 173,099 389,534 2.2504
09 Oct 2018 06:30:16 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 161,579 363,300 2.2484
08 Oct 2018 23:01:39 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 150,059 337,559 2.2495
08 Oct 2018 14:47:43 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 138,539 312,028 2.2523
08 Oct 2018 07:03:58 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 127,019 286,148 2.2528
07 Oct 2018 23:35:20 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 115,499 259,759 2.2490
07 Oct 2018 15:11:31 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 103,979 233,516 2.2458
07 Oct 2018 07:32:54 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 92,459 206,728 2.2359
06 Oct 2018 22:33:56 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 80,939 180,042 2.2244
06 Oct 2018 14:08:52 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 69,419 154,236 2.2218
06 Oct 2018 05:44:34 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 57,899 127,882 2.2087
05 Oct 2018 21:26:05 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 46,379 101,508 2.1887
05 Oct 2018 15:28:21 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 34,859 75,852 2.1760
05 Oct 2018 05:28:35 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 23,339 49,659 2.1277
04 Oct 2018 21:40:12 1346937 21329678 wah2_pnw25_pb8j_198609_28_754_011622920_2 11,819 24,382 2.0629


©2024 climateprediction.net