Task 13738015

Name	hadcm3n_y86s_1940_40_007608043_2
Workunit	7786173
Created	6 Dec 2011, 16:00:12 UTC
Sent	6 Dec 2011, 16:09:54 UTC
Report deadline	6 Mar 2012, 23:37:05 UTC
Received	9 Jun 2012, 3:53:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1053862
Run time	5 days 10 hours 50 min 23 sec
CPU time	4 days 21 hours 53 min 31 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> ile Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 76170000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-exported- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 6aab0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 75ee0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 7.0.6002.18005 ModLoad: 6b1d0000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 6a8c0000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 70840000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18051 ModLoad: 763b0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 75a70000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 76560000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-exported- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 74d50000 0000c000 C:\Windows\system32\dwmapi.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : dwmapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 75d40000 00029000 C:\Windows\system32\imagehlp.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 76540000 0001e000 C:\Windows\system32\IMM32.DLL (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : imm32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 774d0000 000c8000 C:\Windows\system32\MSCTF.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 75a60000 00009000 C:\Windows\system32\LPK.DLL (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : lpk.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18051 ModLoad: 75cc0000 0007d000 C:\Windows\system32\USP10.dll (1.626.6002.18244) (-exported- Symbols Loaded) Linked PDB Filename : usp10.pdb File Version : 1.0626.6002.18244 (vistasp2_gdr.100416-0345) Company Name : Microsoft Corporation Product Name : Microsoft(R) Uniscribe Unicode script processor Product Version : 1.0626.6002.18244 ModLoad: 6a680000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (-nosymbols- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 6af70000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (-nosymbols- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 73f00000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (-nosymbols- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 74d40000 00008000 C:\Windows\system32\version.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 134, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 127792, QuotaPeakPagedPoolUsage: 127792 QuotaNonPagedPoolUsage: 2976, QuotaPeakNonPagedPoolUsage: 2976 - Virtual Memory Usage - VirtualSize: 906768384, PeakVirtualSize: 906768384 - Pagefile Usage - PagefileUsage: 33755136, PeakPagefileUsage: 107069440 - Working Set Size - WorkingSetSize: 4108288, PeakWorkingSetSize: 4108288, PageFaultCount: 1006 * Dump of thread ID 8004 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 156001.000000, Wait Time: 69290.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x772FFC56 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=772ffc56 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!RaiseException+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 001a2539 1132faec 001a24c1 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7730d309 7ffda000 1132ffd4 773e16c3 7ffda000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 773e16c3 7ffda000 6664d5b0 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 1132ffd4 773e1696 007bde7f 7ffda000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffda000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 8012 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 69289.000000 - Registers - eax=00026161 ebx=00000000 ecx=00000000 edx=00000000 esi=3411ff48 edi=00000000 eip=77405ca4 esp=3411ff04 ebp=3411ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3411ff6c 772c1c6c 00000064 00000000 3411ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3411ff7c 00749924 00000064 3411ff94 7730d309 00000000 kernel32!Sleep+0x0 3411ff88 7730d309 00000000 3411ffd4 773e16c3 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3411ff94 773e16c3 00000000 4347d5b0 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3411ffd4 773e1696 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3411ffec 00000000 00749910 00000000 00000000 34630000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7732, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x772FFC56 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 12/20/11 23:30:58 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_y86s_1940_40_007608043;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 773a0000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18327) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 772c0000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18449) (-exported- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6002.18449 (vistasp2_gdr.110412-0338) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18449 ModLoad: 75e40000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 75d70000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 766c0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 76170000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-exported- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 6aab0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 75ee0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 7.0.6002.18005 ModLoad: 6b1d0000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 6a8c0000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 70840000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18051 ModLoad: 763b0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 75a70000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 76560000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-exported- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 74d50000 0000c000 C:\Windows\system32\dwmapi.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : dwmapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 75d40000 00029000 C:\Windows\system32\imagehlp.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 76540000 0001e000 C:\Windows\system32\IMM32.DLL (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : imm32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 774d0000 000c8000 C:\Windows\system32\MSCTF.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 75a60000 00009000 C:\Windows\system32\LPK.DLL (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : lpk.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18051 ModLoad: 75cc0000 0007d000 C:\Windows\system32\USP10.dll (1.626.6002.18244) (-exported- Symbols Loaded) Linked PDB Filename : usp10.pdb File Version : 1.0626.6002.18244 (vistasp2_gdr.100416-0345) Company Name : Microsoft Corporation Product Name : Microsoft(R) Uniscribe Unicode script processor Product Version : 1.0626.6002.18244 ModLoad: 6a680000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 6af70000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 73f00000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 74d40000 00008000 C:\Windows\system32\version.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 127, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 127792, QuotaPeakPagedPoolUsage: 127792 QuotaNonPagedPoolUsage: 2976, QuotaPeakNonPagedPoolUsage: 2976 - Virtual Memory Usage - VirtualSize: 906768384, PeakVirtualSize: 906768384 - Pagefile Usage - PagefileUsage: 33751040, PeakPagefileUsage: 107065344 - Working Set Size - WorkingSetSize: 4075520, PeakWorkingSetSize: 4075520, PageFaultCount: 998 * Dump of thread ID 5076 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156001.000000, User Time: 0.000000, Wait Time: 71846.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x772FFC56 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=772ffc56 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!RaiseException+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00122539 1132faec 001224c1 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7730d309 7ffdf000 1132ffd4 773e16c3 7ffdf000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 773e16c3 7ffdf000 6665b616 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 1132ffd4 773e1696 007bde7f 7ffdf000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffdf000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 4916 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 71846.000000 - Registers - eax=00026161 ebx=00000000 ecx=00000000 edx=00000000 esi=3406ff48 edi=00000000 eip=77405ca4 esp=3406ff04 ebp=3406ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3406ff6c 772c1c6c 00000064 00000000 3406ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3406ff7c 00749924 00000064 3406ff94 7730d309 00000000 kernel32!Sleep+0x0 3406ff88 7730d309 00000000 3406ffd4 773e16c3 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3406ff94 773e16c3 00000000 4351b616 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3406ffd4 773e1696 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3406ffec 00000000 00749910 00000000 00000000 34580000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7732, iMonCtr=1 Model crash detected, will try to restart... 23:31:31 (7732): No heartbeat from core client for 30 sec - exiting 23:31:32 (7732): No heartbeat from core client for 30 sec - exiting 23:31:33 (7732): No heartbeat from core client for 30 sec - exiting 23:31:34 (7732): No heartbeat from core client for 30 sec - exiting 23:31:35 (7732): No heartbeat from core client for 30 sec - exiting 23:31:36 (7732): No heartbeat from core client for 30 sec - exiting 23:31:37 (7732): No heartbeat from core client for 30 sec - exiting 23:31:38 (7732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:35 (6092): No heartbeat from core client for 30 sec - exiting 21:55:37 (6092): No heartbeat from core client for 30 sec - exiting 21:55:38 (6092): No heartbeat from core client for 30 sec - exiting 21:55:39 (6092): No heartbeat from core client for 30 sec - exiting 21:55:40 (6092): No heartbeat from core client for 30 sec - exiting 21:55:41 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:42 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:58:57 (5820): Can't acquire lockfile (32) - waiting 35s 21:58:59 (2156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1236, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:47:48 (5804): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:18:32 (4556): Can't acquire lockfile (32) - waiting 35s Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:13:32 (5104): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6824, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:26:19 (4344): No heartbeat from core client for 30 sec - exiting 00:26:20 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:53:13 (5052): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:51:22 (4880): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7232, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x767AFC56 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 02/05/12 23:44:36 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_y86s_1940_40_007608043;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 776c0000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 76770000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18449) (-exported- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 75f30000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 77810000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 774a0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18005 ModLoad: 76580000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-nosymbols- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 ModLoad: 61890000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 76850000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 7.0.6002.18005 ModLoad: 69550000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 62a50000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 ModLoad: 6c5d0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6002.18051 ModLoad: 761a0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 76350000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 77570000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-nosymbols- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 6.0.6000.16386 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 136, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 127792, QuotaPeakPagedPoolUsage: 127792 QuotaNonPagedPoolUsage: 2888, QuotaPeakNonPagedPoolUsage: 2888 - Virtual Memory Usage - VirtualSize: 884740096, PeakVirtualSize: 884740096 - Pagefile Usage - PagefileUsage: 11702272, PeakPagefileUsage: 16711680 - Working Set Size - WorkingSetSize: 4100096, PeakWorkingSetSize: 4100096, PageFaultCount: 1004 * Dump of thread ID 524 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156001.000000, User Time: 0.000000, Wait Time: 176990.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x767AFC56 - Registers - eax=1132f9cc ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=767afc56 esp=1132f9cc ebp=1132fa1c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1132fa1c 007bfa90 e06d7363 00000001 00000003 1132fa48 kernel32!RaiseException+0x0 1132fa54 007bbbcb 1132fa64 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa70 00406e6e 00500000 004083ee 00000000 003f0000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00182539 1132faec 001824c1 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 767bd309 7ffdf000 1132ffd4 77701603 7ffdf000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 77701603 7ffdf000 66608f22 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 1132ffd4 777015d6 007bde7f 7ffdf000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffdf000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 1060 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 176989.000000 - Registers - eax=0000003c ebx=00000000 ecx=00000069 edx=00000001 esi=3428ff48 edi=00000000 eip=77725cd4 esp=3428ff04 ebp=3428ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3428ff6c 76771c6c 00000064 00000000 3428ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3428ff7c 00749924 00000064 3428ff94 767bd309 00000000 kernel32!Sleep+0x0 3428ff88 767bd309 00000000 3428ffd4 77701603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3428ff94 77701603 00000000 437a8f22 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3428ffd4 777015d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3428ffec 00000000 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7232, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x767AFC56 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 02/05/12 23:44:43 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1455 LoadLibraryA( dbghelp.dll ): GetLastError = 1455 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 132, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 127792, QuotaPeakPagedPoolUsage: 127792 QuotaNonPagedPoolUsage: 2936, QuotaPeakNonPagedPoolUsage: 2936 - Virtual Memory Usage - VirtualSize: 889987072, PeakVirtualSize: 889987072 - Pagefile Usage - PagefileUsage: 16957440, PeakPagefileUsage: 33525760 - Working Set Size - WorkingSetSize: 4083712, PeakWorkingSetSize: 4083712, PageFaultCount: 1000 * Dump of thread ID 5096 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 177464.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x767AFC56 * Dump of thread ID 6468 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 177464.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7232, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:18:22 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:30:01 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:02:35 (5672): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... 22:43:07 (6136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:14:14 (6056): No heartbeat from core client for 30 sec - exiting 22:14:15 (6056): No heartbeat from core client for 30 sec - exiting 22:14:16 (6056): No heartbeat from core client for 30 sec - exiting 22:14:17 (6056): No heartbeat from core client for 30 sec - exiting 22:14:18 (6056): No heartbeat from core client for 30 sec - exiting 22:14:19 (6056): No heartbeat from core client for 30 sec - exiting 22:14:20 (6056): No heartbeat from core client for 30 sec - exiting 22:14:21 (6056): No heartbeat from core client for 30 sec - exiting 22:14:22 (6056): No heartbeat from core client for 30 sec - exiting 22:14:23 (6056): No heartbeat from core client for 30 sec - exiting 22:14:24 (6056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:05:27 (352): No heartbeat from core client for 30 sec - exiting 23:05:28 (352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:56:09 (5124): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:24:06 (4356): No heartbeat from core client for 30 sec - exiting 01:24:07 (4356): No heartbeat from core client for 30 sec - exiting 01:24:08 (4356): No heartbeat from core client for 30 sec - exiting 01:24:10 (4356): No heartbeat from core client for 30 sec - exiting 01:24:11 (4356): No heartbeat from core client for 30 sec - exiting 01:24:12 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:46:03 (2744): No heartbeat from core client for 30 sec - exiting 23:46:04 (2744): No heartbeat from core client for 30 sec - exiting 23:46:05 (2744): No heartbeat from core client for 30 sec - exiting 23:46:06 (2744): No heartbeat from core client for 30 sec - exiting 23:46:07 (2744): No heartbeat from core client for 30 sec - exiting 23:46:08 (2744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:46:09 (2744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:18:00 (4880): No heartbeat from core client for 30 sec - exiting 21:18:01 (4880): No heartbeat from core client for 30 sec - exiting 21:18:02 (4880): No heartbeat from core client for 30 sec - exiting 21:18:03 (4880): No heartbeat from core client for 30 sec - exiting 21:18:04 (4880): No heartbeat from core client for 30 sec - exiting 21:18:05 (4880): No heartbeat from core client for 30 sec - exiting 21:18:06 (4880): No heartbeat from core client for 30 sec - exiting 21:18:07 (4880): No heartbeat from core client for 30 sec - exiting 21:18:08 (4880): No heartbeat from core client for 30 sec - exiting 21:18:09 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:28:23 (4228): No heartbeat from core client for 30 sec - exiting 00:28:24 (4228): No heartbeat from core client for 30 sec - exiting 00:28:25 (4228): No heartbeat from core client for 30 sec - exiting 00:28:26 (4228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:39:48 (4972): No heartbeat from core client for 30 sec - exiting 01:39:50 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:37:04 (5784): No heartbeat from core client for 30 sec - exiting 13:37:05 (5784): No heartbeat from core client for 30 sec - exiting 13:37:06 (5784): No heartbeat from core client for 30 sec - exiting 13:37:07 (5784): No heartbeat from core client for 30 sec - exiting 13:37:08 (5784): No heartbeat from core client for 30 sec - exiting 13:37:09 (5784): No heartbeat from core client for 30 sec - exiting 13:37:10 (5784): No heartbeat from core client for 30 sec - exiting 13:37:11 (5784): No heartbeat from core client for 30 sec - exiting 13:37:12 (5784): No heartbeat from core client for 30 sec - exiting 13:37:13 (5784): No heartbeat from core client for 30 sec - exiting 13:37:15 (5784): No heartbeat from core client for 30 sec - exiting 13:37:16 (5784): No heartbeat from core client for 30 sec - exiting 13:37:17 (5784): No heartbeat from core client for 30 sec - exiting 13:37:18 (5784): No heartbeat from core client for 30 sec - exiting 13:37:19 (5784): No heartbeat from core client for 30 sec - exiting 13:37:20 (5784): No heartbeat from core client for 30 sec - exiting 13:37:21 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:40:08 (4948): No heartbeat from core client for 30 sec - exiting 02:40:09 (4948): No heartbeat from core client for 30 sec - exiting 02:40:10 (4948): No heartbeat from core client for 30 sec - exiting 02:40:11 (4948): No heartbeat from core client for 30 sec - exiting forrtl: ANZXªÛ³êÜµ½B CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2764, selfPID=2764, iMonCtr=1 11:51:03 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:04 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:45:00 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:00:56 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:38 (2704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:48:05 (2796): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:51:30 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:37:19 (5980): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:10:35 (3968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:07:26 (1512): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:23:43 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:07:53 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:53:44 (3416): No heartbeat from core client for 30 sec - exiting 23:53:45 (3416): No heartbeat from core client for 30 sec - exiting 23:53:46 (3416): No heartbeat from core client for 30 sec - exiting 23:53:47 (3416): No heartbeat from core client for 30 sec - exiting 23:53:48 (3416): No heartbeat from core client for 30 sec - exiting 23:53:49 (3416): No heartbeat from core client for 30 sec - exiting 23:53:50 (3416): No heartbeat from core client for 30 sec - exiting 23:53:51 (3416): No heartbeat from core client for 30 sec - exiting 23:53:52 (3416): No heartbeat from core client for 30 sec - exiting 23:53:53 (3416): No heartbeat from core client for 30 sec - exiting 23:53:54 (3416): No heartbeat from core client for 30 sec - exiting 23:53:55 (3416): No heartbeat from core client for 30 sec - exiting 23:53:56 (3416): No heartbeat from core client for 30 sec - exiting 23:53:57 (3416): No heartbeat from core client for 30 sec - exiting 23:53:58 (3416): No heartbeat from core client for 30 sec - exiting 23:53:59 (3416): No heartbeat from core client for 30 sec - exiting 23:54:00 (3416): No heartbeat from core client for 30 sec - exiting 23:54:01 (3416): No heartbeat from core client for 30 sec - exiting 23:54:02 (3416): No heartbeat from core client for 30 sec - exiting 23:54:03 (3416): No heartbeat from core client for 30 sec - exiting 23:54:04 (3416): No heartbeat from core client for 30 sec - exiting 23:54:05 (3416): No heartbeat from core client for 30 sec - exiting 23:54:06 (3416): No heartbeat from core client for 30 sec - exiting 23:54:07 (3416): No heartbeat from core client for 30 sec - exiting 23:54:08 (3416): No heartbeat from core client for 30 sec - exiting 23:54:09 (3416): No heartbeat from core client for 30 sec - exiting 23:54:10 (3416): No heartbeat from core client for 30 sec - exiting 23:54:11 (3416): No heartbeat from core client for 30 sec - exiting 23:54:12 (3416): No heartbeat from core client for 30 sec - exiting 23:54:13 (3416): No heartbeat from core client for 30 sec - exiting 23:54:14 (3416): No heartbeat from core client for 30 sec - exiting 23:54:15 (3416): No heartbeat from core client for 30 sec - exiting 23:54:16 (3416): No heartbeat from core client for 30 sec - exiting 23:54:17 (3416): No heartbeat from core client for 30 sec - exiting 23:54:18 (3416): No heartbeat from core client for 30 sec - exiting 23:54:19 (3416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:52:35 (2296): No heartbeat from core client for 30 sec - exiting 23:52:36 (2296): No heartbeat from core client for 30 sec - exiting 23:52:37 (2296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:35:43 (5936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:13:26 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:33 (1388): Can't acquire lockfile (32) - waiting 35s 02:39:45 (3372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:00:27 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:19 (3828): Can't acquire lockfile (32) - waiting 35s 11:01:42 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Jun 2012 03:56:22	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	259,200	424,398	1.6373
29 Apr 2012 01:27:23	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	233,280	385,781	1.6537
06 Mar 2012 22:28:16	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	207,360	344,472	1.6612
06 Feb 2012 12:50:59	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	181,440	299,882	1.6528
29 Jan 2012 14:13:33	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	155,520	251,640	1.6181
19 Jan 2012 18:24:23	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	129,600	206,469	1.5931
09 Jan 2012 22:35:54	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	103,680	164,712	1.5887
07 Jan 2012 22:17:34	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	77,760	124,716	1.6039
23 Dec 2011 23:08:03	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	51,840	82,057	1.5829
23 Dec 2011 15:52:44	1053862	13738015	hadcm3n_y86s_1940_40_007608043_2	25,920	41,014	1.5823