climateprediction.net home page
Task 19665592

Task 19665592

Name hadam3p_pnw_j2lk_201412_12_404_010572790_0
Workunit 10572790
Created 16 May 2016, 11:23:28 UTC
Sent 17 May 2016, 5:53:19 UTC
Report deadline 29 Apr 2017, 11:13:19 UTC
Received 23 May 2016, 10:41:49 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1296727
Run time 1 days 17 hours 31 min 16 sec
CPU time 1 days 17 hours 25 min 10 sec
Validate state Workunit error - check skipped
Credit 3,010.29
Device peak FLOPS 4.37 GFLOPS
Application version UK Met Office HadAM3P-HadRM3P Pacific North West v7.27
windows_intelx86
Stderr
<core_client_version>7.4.42</core_client_version>
<![CDATA[
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x0171D3AC

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.5.1


Dump Timestamp    : 05/17/16 17:46:40
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\4;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 000000000018e000 C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_pnw_7.27_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077c00000 000000000017b000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000076f90000 00000000000e0000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000076ae0000 000000000017e000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000747a0000 0000000000092000 C:\WINDOWS\system32\apphelp.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : apphelp.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000077170000 0000000000147000 C:\WINDOWS\SYSTEM32\USER32.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075130000 000000000014f000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.10586.306 (th2_release_sec.160422-1850)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.306

ModLoad: 0000000077b30000 000000000007b000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.2.10586.63) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000773f0000 00000000000be000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 7.0.10586.0

ModLoad: 0000000076ea0000 0000000000044000 C:\WINDOWS\SYSTEM32\sechost.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000753e0000 00000000000ad000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074850000 000000000001e000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074840000 000000000000a000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075650000 0000000000058000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000072060000 0000000000028000 C:\WINDOWS\SYSTEM32\ntmarta.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000055e30000 0000000000071000 C:\WINDOWS\SYSTEM32\efswrt.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : efswrt.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074ac0000 000000000008d000 C:\WINDOWS\SYSTEM32\SHCORE.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075490000 00000000001bd000 C:\WINDOWS\SYSTEM32\combase.dll (6.2.10586.103) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000773d0000 000000000000c000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

Get Product Name Failed.
ModLoad: 0000000076ef0000 0000000000092000 C:\WINDOWS\SYSTEM32\OLEAUT32.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 10.0.10586.0
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 10.0.10586.0

ModLoad: 0000000070c50000 00000000000c8000 C:\WINDOWS\SYSTEM32\wintypes.dll (6.2.10586.103) (-exported- Symbols Loaded)
    Linked PDB Filename   : WinTypes.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 000000005d280000 0000000000049000 C:\WINDOWS\SYSTEM32\edputil.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : edputil.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075390000 0000000000045000 C:\WINDOWS\SYSTEM32\shlwapi.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000070b10000 000000000013f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073f80000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : MicrosoftR WindowsR Operating System
    Product Version       : 10.0.10586.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 18083, Write: 0, Other 69192

- I/O Transfers Counters -
Read: 0, Write: 3310, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 111544, QuotaPeakPagedPoolUsage: 127280
QuotaNonPagedPoolUsage: 8824, QuotaPeakNonPagedPoolUsage: 9768

- Virtual Memory Usage -
VirtualSize: 64225280, PeakVirtualSize: 72495104

- Pagefile Usage -
PagefileUsage: 2748416, PeakPagefileUsage: 3436544

- Working Set Size -
WorkingSetSize: 31289344, PeakWorkingSetSize: 32702464, PageFaultCount: 9794

*** Dump of thread ID 10176 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2086277.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x0171D3AC

- Registers -
eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=0164d020 edi=00000000
eip=00462b87 esp=00a2fe7c ebp=00a2fe80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216

- Callstack -
ChildEBP RetAddr  Args to Child
00a2fe80 00447d6f 0164d020 00000000 0164d020 fffffffe hadam3p_pnw_7.27_windows_intelx!+0x0 
00a2fe98 0044f568 0164d020 00000000 005597a4 00a2fee8 hadam3p_pnw_7.27_windows_intelx!+0x0 
00a2feac 0048374c 00000015 723bca02 00000006 004836bc hadam3p_pnw_7.27_windows_intelx!+0x0 
00a2fee8 76bfa98f 00000006 2a5599e7 76bfa7d0 76bfa7d0 hadam3p_pnw_7.27_windows_intelx!+0x0 
00a2ff80 76fa38f4 80000006 76fa38d0 af7e9edd 00a2ffdc KERNELBASE!CtrlRoutine+0x0 
00a2ff94 77c65de3 80000006 fcafd3e7 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00a2ffdc 77c65dae ffffffff 77c8b7d1 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
00a2ffec 00000000 76bfa7d0 80000006 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 8956 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 5312500.000000, User Time: 32968750.000000, Wait Time: 2086266.000000

- Registers -
eax=00000000 ebx=0019f9d0 ecx=00000000 edx=00000000 esi=00000000 edi=00002a67
eip=77c76f1c esp=0019f990 ebp=0019f9f4
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000216

- Callstack -
ChildEBP RetAddr  Args to Child
0019f9f4 76b9a6cf 000000c8 00000000 0019fa18 004689c1 ntdll!ZwDelayExecution+0x0 
0019fa04 004689c1 000000c8 000000c8 00000000 0019fa50 KERNELBASE!Sleep+0x0 
0019fa18 00403ddc a0000000 027f9999 004cf858 004cfb88 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fa50 00404d76 00000000 7280c9aa 00484e6e 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fd40 00404f67 0019fef0 00404fe5 004d02a8 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fd48 00404fe5 004d02a8 0019fd60 00733440 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fef0 00484e1b 00400000 00000000 00733485 0000000a hadam3p_pnw_7.27_windows_intelx!+0x0 
0019ff80 76fa38f4 003d5000 76fa38d0 afc59edd 0019ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 
0019ff94 77c65de3 003d5000 fc14d3e7 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77c65dae ffffffff 77c8b7d1 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0019ffec 00000000 00484e6e 003d5000 00000000 78746341 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 8480 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2086276.000000

- Registers -
eax=c0000022 ebx=0163fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77c76f1c esp=0163fcdc ebp=0163fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0163fd40 76b9a6cf 00000064 00000000 0163ff80 00477e94 ntdll!ZwDelayExecution+0x0 
0163fd50 00477e94 00000064 00477e70 00477e70 00000000 KERNELBASE!Sleep+0x0 
0163ff80 76fa38f4 00000000 76fa38d0 aebf9edd 0163ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 
0163ff94 77c65de3 00000000 fd6ed3e7 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0163ffdc 77c65dae ffffffff 77c8b7d1 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0163ffec 00000000 00477e70 00000000 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 11216 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2086270.000000

- Registers -
eax=00000000 ebx=00736540 ecx=00000000 edx=00000000 esi=00736380 edi=00733c38
eip=77c7876c esp=0092fdc8 ebp=0092ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0092ff80 76fa38f4 00733c38 76fa38d0 af4e9edd 0092ffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
0092ff94 77c65de3 00733c38 fc9fd3e7 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0092ffdc 77c65dae ffffffff 77c8b7d1 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0092ffec 00000000 77c2c6d0 00733c38 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9616, selfPID=4152, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9616, selfPID=9616, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
15:11:13 (4920): start_timer_thread(): CreateThread() failed, errno 0
15:11:14 (5012): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4968, selfPID=5064, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4968, selfPID=4968, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5532, selfPID=5532, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5532, selfPID=3176, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Leaving CPDN_Main::Monitor...
17:38:57 (9164): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 May 2016 09:42:49 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 138,539 149,027 1.0757
23 May 2016 04:04:01 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 127,019 136,336 1.0734
20 May 2016 09:48:43 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 115,499 123,604 1.0702
20 May 2016 06:12:13 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 103,979 110,863 1.0662
20 May 2016 02:37:07 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 92,459 97,979 1.0597
19 May 2016 23:08:10 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 80,939 85,223 1.0529
19 May 2016 17:41:56 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 69,419 73,400 1.0573
19 May 2016 14:23:49 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 57,899 61,670 1.0651
19 May 2016 02:14:38 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 46,379 49,637 1.0702
18 May 2016 08:04:56 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 34,859 37,264 1.0690
18 May 2016 04:30:23 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 23,339 25,332 1.0854
18 May 2016 01:12:58 1296727 19665592 hadam3p_pnw_j2lk_201412_12_404_010572790_0 11,819 13,396 1.1334


©2024 climateprediction.net