Task 15495029

Name	hadcm3n_ygrp_1980_40_008244305_4
Workunit	8399429
Created	21 Dec 2012, 13:51:56 UTC
Sent	21 Dec 2012, 13:58:15 UTC
Report deadline	22 Mar 2013, 21:25:26 UTC
Received	26 Feb 2013, 17:10:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-529697949 (0xE06D7363) Unknown error code
Computer ID	825978
Run time	9 days 6 hours 33 min 9 sec
CPU time	9 days 6 hours 33 min 9 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.4.6</core_client_version> <![CDATA[ <message> - exit code -529697949 (0xe06d7363) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1292, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3472, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4560, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=608, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6176, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1312, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit requWorker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3764, selfPID=3764, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=676, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 02/26/13 16:37:29 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll * Dump of the Process Statistics: * - I/O Operations Counters - Read: 366629, Write: 0, Other 110875 - I/O Transfers Counters - Read: 0, Write: 16775, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 91672, QuotaPeakPagedPoolUsage: 98256 QuotaNonPagedPoolUsage: 2688, QuotaPeakNonPagedPoolUsage: 3192 - Virtual Memory Usage - VirtualSize: 52420608, PeakVirtualSize: 134684672 - Pagefile Usage - PagefileUsage: 3870720, PeakPagefileUsage: 85323776 - Working Set Size - WorkingSetSize: 9326592, PeakWorkingSetSize: 90497024, PageFaultCount: 87544 * Dump of thread ID 5048 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 81900528.000000, User Time: 309661984.000000, Wait Time: 1709614.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 * Dump of thread ID 5416 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 156001.000000, User Time: 468003.000000, Wait Time: 1709612.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Feb 2013 15:40:52	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	259,200	801,182	3.0910
24 Feb 2013 20:16:22	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	233,280	722,123	3.0955
21 Feb 2013 14:00:26	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	207,360	642,462	3.0983
18 Feb 2013 20:39:10	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	181,440	562,803	3.1019
16 Feb 2013 19:42:05	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	155,520	483,015	3.1058
14 Feb 2013 12:01:23	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	129,600	404,478	3.1210
12 Feb 2013 13:55:41	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	103,680	324,377	3.1286
08 Feb 2013 19:45:26	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	77,760	244,534	3.1447
19 Jan 2013 21:11:38	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	51,840	164,163	3.1667
12 Jan 2013 16:40:52	825978	15495029	hadcm3n_ygrp_1980_40_008244305_4	25,920	82,439	3.1805