Task 12818968

Name	hadcm3n_p1hv_1900_40_007219523_1
Workunit	7417763
Created	26 Apr 2011, 15:17:51 UTC
Sent	3 May 2011, 21:53:07 UTC
Report deadline	3 Aug 2011, 5:20:18 UTC
Received	10 Jun 2011, 10:25:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1147345
Run time	16 days 18 hours 5 min 9 sec
CPU time	16 days 4 hours 30 min 46 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	1.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3748, iMonCtr=14:44:36 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1176, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2476, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5040, iMonCtr=1 Model crash detected, will try to restart... 14:57:32 (3964): No heartbeat from core client for 30 sec - exiting 14:57:35 (3964): No heartbeat from core client for 30 sec - exiting 14:57:36 (3964): No heartbeat from core client for 30 sec - exiting 14:57:37 (3964): No heartbeat from core client for 30 sec - exiting 14:57:38 (3964): No heartbeat from core client for 30 sec - exiting 14:57:39 (3964): No heartbeat from core client for 30 sec - exiting 14:57:40 (3964): No heartbeat from core client for 30 sec - exiting 14:57:41 (3964): No heartbeat from core client for 30 sec - exiting 14:57:42 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:44 (3964): No heartbeat from core client for 30 sec - exiting Ocean Restart file copy failed on p1hvko.daa76d0 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process isCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:13:58 (1356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=944, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:27:59 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jun 2011 07:48:28	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	570,240	1,397,585	2.4509
07 Jun 2011 08:59:31	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	544,320	1,329,906	2.4432
03 Jun 2011 01:46:05	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	518,400	1,269,186	2.4483
02 Jun 2011 07:42:28	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	492,480	1,208,804	2.4545
31 May 2011 04:57:15	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	466,560	1,148,067	2.4607
30 May 2011 12:05:01	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	440,640	1,087,620	2.4683
26 May 2011 23:51:24	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	414,720	1,026,717	2.4757
26 May 2011 01:31:23	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	388,800	965,991	2.4845
25 May 2011 06:21:35	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	362,880	905,409	2.4951
24 May 2011 13:20:33	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	336,960	844,977	2.5076
23 May 2011 17:13:56	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	311,040	784,550	2.5223
23 May 2011 00:08:55	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	285,120	723,834	2.5387
22 May 2011 07:16:30	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	259,200	663,376	2.5593
21 May 2011 14:21:50	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	233,280	602,812	2.5841
20 May 2011 20:26:12	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	207,360	538,690	2.5978
20 May 2011 00:26:31	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	181,440	470,833	2.5950
13 May 2011 06:37:43	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	155,520	402,953	2.5910
12 May 2011 11:50:05	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	129,600	336,471	2.5962
11 May 2011 17:18:03	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	103,680	270,036	2.6045
09 May 2011 23:04:28	1147345	12818968	hadcm3n_p1hv_1900_40_007219523_1	77,760	202,266	2.6012