Task 17357738

Name	hadcm3n_xcyt_1940_40_009153099_0
Workunit	9283435
Created	6 Nov 2014, 15:50:48 UTC
Sent	6 Nov 2014, 19:16:15 UTC
Report deadline	6 Feb 2015, 2:43:26 UTC
Received	2 Dec 2014, 0:14:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-529697949 (0xE06D7363) Unknown error code
Computer ID	1411061
Run time	9 days 18 hours 11 min 37 sec
CPU time	7 days 22 hours 5 min 13 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.55 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> (unknown error) - exit code -529697949 (0xe06d7363) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:03:53 (21388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:10:42 (6624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:59 (7088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:57:26 (448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:29:57 (1284): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:11:20 (20848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:51:01 (6188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:56:39 (8824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:37 (6752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:32:28 (7260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:54:22 (6812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x778C2F71 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 12/02/14 10:44:16 Install Directory : Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_xcyt_1940_40_009153099;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00111000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77aa0000 00167000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.3.9600.17278) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 75840000 00140000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.3.9600.17056) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 778b0000 000d0000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.3.9600.17278) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 764e0000 0014c000 C:\WINDOWS\SYSTEM32\USER32.dll (6.3.9600.17344) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 75bb0000 00078000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.3.9600.17031) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 75730000 00107000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.3.9600.17246) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 6.3.9600.17246 (winblue_gdr.140801-1518) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17246 ModLoad: 76230000 000be000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.9600.16384) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.9600.16384 ModLoad: 762f0000 0003e000 C:\WINDOWS\SYSTEM32\sechost.dll (6.3.9600.16384) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 77800000 000b0000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.3.9600.17216) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 754b0000 0001d000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.3.9600.16408) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.3.9600.16408 (winblue_gdr.130920-1803) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16408 ModLoad: 754a0000 00009000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.3.9600.16384) (-exported- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 75440000 00052000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.3.9600.17120) (-exported- Symbols Loaded) Linked PDB Filename : bcryptprimitives.pdb File Version : 6.3.9600.17120 (winblue_gdr.140509-1516) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17120 ModLoad: 76330000 00025000 C:\WINDOWS\system32\IMM32.DLL (6.3.9600.17031) (-exported- Symbols Loaded) Linked PDB Filename : wimm32.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 759d0000 000f7000 C:\WINDOWS\SYSTEM32\MSCTF.dll (6.3.9600.17031) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 75380000 00025000 C:\WINDOWS\SYSTEM32\ntmarta.dll (6.3.9600.16384) (-exported- Symbols Loaded) Linked PDB Filename : ntmarta.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 71420000 000a4000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_se_6.07_windows_intelx86.dll (-exported- Symbols Loaded) Linked PDB Filename : d:\cpdn\cpdnboinc\hadcm3\test\projects\reqs.comlab.ox.ac.uk_cpdn\hadcm3n_se_6.07_windows_intelx86.pdb ModLoad: 70730000 00148000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.3.9600.16520) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.3.9600.16520 (winblue_gdr.140127-0329) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16520 ModLoad: 75430000 00008000 C:\WINDOWS\SYSTEM32\version.dll (6.3.9600.16384) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 7748 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x778C2F71 - Registers - eax=0018f220 ebx=02b80048 ecx=00000003 edx=00000000 esi=714ba484 edi=013afa00 eip=778c2f71 esp=0018f220 ebp=0018f27c cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 0018f27c 7147e730 e06d7363 00000001 00000003 0018f2a8 KERNELBASE!RaiseException+0x0 0018f2b4 7147bcdc 0018f2c4 714b1fe0 714a13f4 714ab230 hadcm3n_se_6.07_windows_intelx8!umtonc+0x0 0018f2d0 7142fa97 04ebe800 00000010 02d73c28 00000000 hadcm3n_se_6.07_windows_intelx8!umtonc+0x0 0018f2e8 714236b8 4bd7fba5 004bb848 004bb710 0018f6e8 hadcm3n_se_6.07_windows_intelx8!+0x0 00000000 00000000 00000000 00000000 00000000 00000000 hadcm3n_se_6.07_windows_intelx8!+0x0 * Dump of thread ID 7140 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Registers - eax=c000000f ebx=01ffff4c ecx=00000000 edx=00000000 esi=00000000 edi=01ffff4c eip=77ada83c esp=01ffff0c ebp=01ffff70 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 01ffff70 778b103a 00000064 00000000 01ffff8c 00413da4 ntdll!ZwDelayExecution+0x0 01ffff80 00413da4 00000064 01ffff98 7585919f 00000000 KERNELBASE!Sleep+0x0 01ffff8c 7585919f 00000000 01ffffdc 77af0bbb 00000000 hadcm3n_6.07_windows_intelx86!+0x0 01ffff98 77af0bbb 00000000 5adf4a2c 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 01ffffdc 77af0b91 ffffffff 77adc9ce 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 01ffffec 00000000 00413d90 00000000 00000000 63646168 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Dec 2014 22:17:08	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	440,640	645,928	1.4659
30 Nov 2014 23:47:43	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	414,720	609,408	1.4694
30 Nov 2014 21:48:50	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	388,800	575,172	1.4794
29 Nov 2014 01:01:45	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	362,880	536,244	1.4777
27 Nov 2014 10:55:17	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	336,960	494,740	1.4682
25 Nov 2014 21:37:11	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	311,040	454,429	1.4610
24 Nov 2014 10:35:33	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	285,120	414,989	1.4555
23 Nov 2014 07:29:31	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	259,200	374,645	1.4454
17 Nov 2014 17:54:49	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	233,280	332,970	1.4273
16 Nov 2014 23:46:18	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	207,360	290,238	1.3997
16 Nov 2014 01:54:49	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	181,440	283,953	1.5650
14 Nov 2014 12:03:15	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	155,520	243,803	1.5677
13 Nov 2014 05:09:30	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	129,600	202,834	1.5651
11 Nov 2014 07:14:40	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	103,680	158,505	1.5288
10 Nov 2014 06:16:13	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	77,760	116,439	1.4974
09 Nov 2014 08:15:26	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	51,840	75,825	1.4627
08 Nov 2014 07:40:22	1336781	17357738	hadcm3n_xcyt_1940_40_009153099_0	25,920	38,454	1.4836