climateprediction.net home page
Task 21535820

Task 21535820

Name wah2_anz50_n4e3_201612_20_794_011768077_0
Workunit 11768077
Created 9 Mar 2019, 9:53:34 UTC
Sent 2 Apr 2019, 8:05:28 UTC
Report deadline 14 Mar 2020, 13:25:28 UTC
Received 11 Apr 2019, 3:59:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1346937
Run time 7 days 21 hours 2 min 42 sec
CPU time 7 days 5 hours 54 min 14 sec
Validate state Invalid
Credit 9,898.90
Device peak FLOPS 4.14 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 231.80 MB
Peak swap size 195.45 MB
Peak disk usage 86.55 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20592, selfPID=2324, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
19:49:04 (2324): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0279A12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 04/10/19 19:58:23
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000076f20000 000000000019c000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17763.404) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074c20000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17763.437) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000076050000 00000000001fa000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17763.404) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075a30000 0000000000199000 C:\WINDOWS\System32\USER32.dll (6.2.17763.168) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074f10000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 00000000766a0000 0000000000023000 C:\WINDOWS\System32\GDI32.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000076250000 0000000000167000 C:\WINDOWS\System32\gdi32full.dll (6.2.17763.437) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.17763.437 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.437

ModLoad: 0000000076d40000 0000000000080000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074990000 0000000000122000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17763.404) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17763.404 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.404

ModLoad: 0000000074910000 000000000007e000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075f90000 00000000000c0000 C:\WINDOWS\System32\msvcrt.dll (7.0.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17763.1

ModLoad: 0000000075c70000 0000000000079000 C:\WINDOWS\System32\sechost.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074830000 00000000000bf000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17763.379) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074590000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074580000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074b20000 0000000000062000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 00000000754b0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 000000006f380000 000000000018f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000072fe0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 688, Write: 0, Other 6887

- I/O Transfers Counters -
Read: 0, Write: 1968, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 93104, QuotaPeakPagedPoolUsage: 93264
QuotaNonPagedPoolUsage: 9032, QuotaPeakNonPagedPoolUsage: 9456

- Virtual Memory Usage -
VirtualSize: 55816192, PeakVirtualSize: 55816192

- Pagefile Usage -
PagefileUsage: 3592192, PeakPagefileUsage: 3592192

- Working Set Size -
WorkingSetSize: 5951488, PeakWorkingSetSize: 5955584, PageFaultCount: 1749

*** Dump of thread ID 7808 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 4531250.000000, User Time: 5156250.000000, Wait Time: 27610.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0279A12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc80 edx=0279a12c esi=0019da38 edi=0019dcb0
eip=0048ee46 esp=0019d9cc ebp=0019dc5c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc5c 0048161c 0019dc80 004b63b8 00000000 0019dcb0 wah2_8.24_windows_intelx86!+0x0 
0019dca0 0044fa52 0279a12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc0 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dcd8 00485ca1 c0000005 0019dd0c 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff70 74c40419 0026e000 74c40400 0019ffdc 76f8662d wah2_8.24_windows_intelx86!+0x0 
0019ff80 76f8662d 0026e000 d32e885b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 76f865fd ffffffff 76fa519e 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 0026e000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 9344 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 26403.000000

- Registers -
eax=00000000 ebx=008a57f0 ecx=00000000 edx=00000000 esi=008a5630 edi=008a42a0
eip=76f9216c esp=007afdb4 ebp=007aff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
007aff70 74c40419 008a42a0 74c40400 007affdc 76f8662d ntdll!NtWaitForWorkViaWorkerFactory+0x0 
007aff80 76f8662d 008a42a0 d34d885b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
007affdc 76f865fd ffffffff 76fa519e 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
007affec 00000000 76f6e230 008a42a0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 3832 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 26403.000000

- Registers -
eax=00000000 ebx=008a6218 ecx=00000000 edx=00000000 esi=008a6058 edi=008a42a0
eip=76f9216c esp=00a9fdb4 ebp=00a9ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
00a9ff70 74c40419 008a42a0 74c40400 00a9ffdc 76f8662d ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00a9ff80 76f8662d 008a42a0 d39e885b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00a9ffdc 76f865fd ffffffff 76fa519e 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00a9ffec 00000000 76f6e230 008a42a0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 15228 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 26403.000000

- Registers -
eax=00000000 ebx=008a6668 ecx=00000000 edx=00000000 esi=008a64a8 edi=008a42a0
eip=76f9216c esp=00b9fdb4 ebp=00b9ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
00b9ff70 74c40419 008a42a0 74c40400 00b9ffdc 76f8662d ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00b9ff80 76f8662d 008a42a0 d38e885b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b9ffdc 76f865fd ffffffff 76fa519e 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b9ffec 00000000 76f6e230 008a42a0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 14220 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 27607.000000

- Registers -
eax=00000000 ebx=026cfd0c ecx=00000000 edx=00000000 esi=00000000 edi=026cfd0c
eip=76f907ec esp=026cfccc ebp=026cfd30
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
026cfd30 7615db6f 00000064 00000000 026cff70 00478b44 ntdll!ZwDelayExecution+0x0 
026cfd40 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
026cff70 74c40419 00000000 74c40400 026cffdc 76f8662d wah2_8.24_windows_intelx86!+0x0 
026cff80 76f8662d 00000000 d15b885b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
026cffdc 76f865fd ffffffff 76fa519e 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
026cffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Apr 2019 02:05:59 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 150,059 624,145 4.1593
10 Apr 2019 09:44:41 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 138,539 575,989 4.1576
09 Apr 2019 19:06:30 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 127,019 528,045 4.1572
09 Apr 2019 04:12:43 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 115,499 480,871 4.1634
08 Apr 2019 16:54:11 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 103,979 435,037 4.1839
07 Apr 2019 21:57:12 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 92,459 385,597 4.1705
07 Apr 2019 04:49:41 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 80,939 335,818 4.1490
06 Apr 2019 11:13:30 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 69,419 286,461 4.1266
05 Apr 2019 19:46:11 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 57,899 236,761 4.0892
05 Apr 2019 04:26:51 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 46,379 189,886 4.0942
04 Apr 2019 11:49:18 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 34,859 142,730 4.0945
03 Apr 2019 20:49:39 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 23,339 94,428 4.0459
03 Apr 2019 05:50:22 1346937 21535820 wah2_anz50_n4e3_201612_20_794_011768077_0 11,819 48,291 4.0859


©2024 climateprediction.net