Task 13540620

Name	hadcm3n_yf8d_1900_40_007517471_0
Workunit	7714946
Created	28 Oct 2011, 12:52:39 UTC
Sent	21 Nov 2011, 21:21:26 UTC
Report deadline	21 Feb 2012, 4:48:37 UTC
Received	3 Dec 2011, 9:45:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1036046
Run time	2 days 3 hours 36 min 11 sec
CPU time	1 days 19 hours 56 min 6 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 18:48:23 (4072): No heartbeat from core client for 30 sec - exiting 18:48:24 (4072): No heartbeat from core client for 30 sec - exiting 18:48:25 (4072): No heartbeat from core client for 30 sec - exiting 18:48:26 (4072): No heartbeat from core client for 30 sec - exiting 18:48:27 (4072): No heartbeat from core client for 30 sec - exiting 18:48:28 (4072): No heartbeat from core client for 30 sec - exiting 18:48:29 (4072): No heartbeat from core client for 30 sec - exiting 18:48:30 (4072): No heartbeat from core client for 30 sec - exiting 18:48:31 (4072): No heartbeat from core client for 30 sec - exiting 18:48:32 (4072): No heartbeat from core client for 30 sec - exiting 18:48:33 (4072): No heartbeat from core client for 30 sec - exiting 18:48:34 (4072): No heartbeat from core client for 30 sec - exiting 18:48:35 (4072): No heartbeat from core client for 30 sec - exiting 18:48:36 (4072): No heartbeat from core client for 30 sec - exiting 18:48:37 (4072): No heartbeat from core client for 30 sec - exiting 18:48:38 (4072): No heartbeat from core client for 30 sec - exiting 18:48:39 (4072): No heartbeat from core client for 30 sec - exiting 18:48:40 (4072): No heartbeat from core client for 30 sec - exiting 18:48:41 (4072): No heartbeat from core client for 30 sec - exiting 18:48:42 (4072): No heartbeat from core client for 30 sec - exiting 18:48:43 (4072): No heartbeat from core client for 30 sec - exiting 18:48:44 (4072): No heartbeat from core client for 30 sec - exiting 18:48:45 (4072): No heartbeat from core client for 30 sec - exiting 18:48:46 (4072): No heartbeat from core client for 30 sec - exiting 18:48:47 (4072): No heartbeat from core client for 30 sec - exiting 18:48:48 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4700, iMonCtr=1 Model crash detected, will try to restart... 06:20:49 (4876): No heartbeat from core client for 30 sec - exiting 06:20:50 (4876): No heartbeat from core client for 30 sec - exiting 06:20:51 (4876): No heartbeat from core client for 30 sec - exiting 06:20:52 (4876): No heartbeat from core client for 30 sec - exiting 06:20:54 (4876): No heartbeat from core client for 30 sec - exiting 06:20:55 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:42 (7356): No heartbeat from core client for 30 sec - exiting 19:38:43 (7356): No heartbeat from core client for 30 sec - exiting 19:38:44 (7356): No heartbeat from core client for 30 sec - exiting 19:38:45 (7356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... 17:29:19 (4160): No heartbeat from core client for 30 sec - exiting 17:29:21 (4160): No heartbeat from core client for 30 sec - exiting 17:29:22 (4160): No heartbeat from core client for 30 sec - exiting 17:29:23 (4160): No heartbeat from core client for 30 sec - exiting 17:29:24 (4160): No heartbeat from core client for 30 sec - exiting 17:29:25 (4160): No heartbeat from core client for 30 sec - exiting 17:29:26 (4160): No heartbeat from core client for 30 sec - exiting 17:29:27 (4160): No heartbeat from core client for 30 sec - exiting 17:29:28 (4160): No heartbeat from core client for 30 sec - exiting 17:29:29 (4160): No heartbeat from core client for 30 sec - exiting 17:29:30 (4160): No heartbeat from core client for 30 sec - exiting 17:29:31 (4160): No heartbeat from core client for 30 sec - exiting 17:29:32 (4160): No heartbeat from core client for 30 sec - exiting 17:29:34 (4160): No heartbeat from core client for 30 sec - exiting 17:29:35 (4160): No heartbeat from core client for 30 sec - exiting 17:29:36 (4160): No heartbeat from core client for 30 sec - exiting 17:29:37 (4160): No heartbeat from core client for 30 sec - exiting 17:29:38 (4160): No heartbeat from core client for 30 sec - exiting 17:29:39 (4160): No heartbeat from core client for 30 sec - exiting 17:29:40 (4160): No heartbeat from core client for 30 sec - exiting 17:29:41 (4160): No heartbeat from core client for 30 sec - exiting 17:29:42 (4160): No heartbeat from core client for 30 sec - exiting 17:29:43 (4160): No heartbeat from core client for 30 sec - exiting 17:29:44 (4160): No heartbeat from core client for 30 sec - exiting 17:29:46 (4160): No heartbeat from core client for 30 sec - exiting 17:29:47 (4160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:15:44 (6000): No heartbeat from core client for 30 sec - exiting 08:15:45 (6000): No heartbeat from core client for 30 sec - exiting 08:15:46 (6000): No heartbeat from core client for 30 sec - exiting 08:15:47 (6000): No heartbeat from core client for 30 sec - exiting 08:15:48 (6000): No heartbeat from core client for 30 sec - exiting 08:15:49 (6000): No heartbeat from core client for 30 sec - exiting 08:15:50 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:31:36 (4684): No heartbeat from core client for 30 sec - exiting 09:31:37 (4684): No heartbeat from core client for 30 sec - exiting 09:31:38 (4684): No heartbeat from core client for 30 sec - exiting 09:31:39 (4684): No heartbeat from core client for 30 sec - exiting 09:31:41 (4684): No heartbeat from core client for 30 sec - exiting 09:31:42 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:18 (4728): No heartbeat from core client for 30 sec - exiting 18:21:19 (4728): No heartbeat from core client for 30 sec - exiting 18:21:20 (4728): No heartbeat from core client for 30 sec - exiting 18:21:21 (4728): No heartbeat from core client for 30 sec - exiting 18:21:22 (4728): No heartbeat from core client for 30 sec - exiting 18:21:23 (4728): No heartbeat from core client for 30 sec - exiting 18:21:24 (4728): No heartbeat from core client for 30 sec - exiting 18:21:25 (4728): No heartbeat from core client for 30 sec - exiting 18:21:27 (4728): No heartbeat from core client for 30 sec - exiting 18:21:28 (4728): No heartbeat from core client for 30 sec - exiting 18:21:29 (4728): No heartbeat from core client for 30 sec - exiting 18:21:30 (4728): No heartbeat from core client for 30 sec - exiting 18:21:31 (4728): No heartbeat from core client for 30 sec - exiting 18:21:32 (4728): No heartbeat from core client for 30 sec - exiting 18:21:33 (4728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5804, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17524, iMonCtr=1 Model crash detected, will try to restart... 06:24:22 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:23 (5760): No heartbeat from core client for 30 sec - exiting 18:45:51 (4768): No heartbeat from core client for 30 sec - exiting 18:45:52 (4768): No heartbeat from core client for 30 sec - exiting 18:45:53 (4768): No heartbeat from core client for 30 sec - exiting 18:45:54 (4768): No heartbeat from core client for 30 sec - exiting 18:45:55 (4768): No heartbeat from core client for 30 sec - exiting 18:45:56 (4768): No heartbeat from core client for 30 sec - exiting 18:45:57 (4768): No heartbeat from core client for 30 sec - exiting 18:45:58 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:33:24 (7712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:25 (7712): No heartbeat from core client for 30 sec - exiting 20:33:26 (7712): No heartbeat from core client for 30 sec - exiting 20:33:27 (7712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 06:25:35 (4484): No heartbeat from core client for 30 sec - exiting 06:25:36 (4484): No heartbeat from core client for 30 sec - exiting 06:25:37 (4484): No heartbeat from core client for 30 sec - exiting 06:25:38 (4484): No heartbeat from core client for 30 sec - exiting 06:25:39 (4484): No heartbeat from core client for 30 sec - exiting 06:25:40 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:41 (4108): No heartbeat from core client for 30 sec - exiting 18:53:42 (4108): No heartbeat from core client for 30 sec - exiting 18:53:43 (4108): No heartbeat from core client for 30 sec - exiting 18:53:44 (4108): No heartbeat from core client for 30 sec - exiting 18:53:45 (4108): No heartbeat from core client for 30 sec - exiting 18:53:47 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:48 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 06:24:58 (2252): No heartbeat from core client for 30 sec - exiting 06:24:59 (2252): No heartbeat from core client for 30 sec - exiting 06:25:00 (2252): No heartbeat from core client for 30 sec - exiting 06:25:02 (2252): No heartbeat from core client for 30 sec - exiting 06:25:03 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:25:04 (2252): No heartbeat from core client for 30 sec - exiting 18:08:55 (3696): No heartbeat from core client for 30 sec - exiting 18:08:56 (3696): No heartbeat from core client for 30 sec - exiting 18:08:58 (3696): No heartbeat from core client for 30 sec - exiting 18:08:59 (3696): No heartbeat from core client for 30 sec - exiting 18:09:00 (3696): No heartbeat from core client for 30 sec - exiting 18:09:01 (3696): No heartbeat from core client for 30 sec - exiting 18:09:02 (3696): No heartbeat from core client for 30 sec - exiting 18:09:03 (3696): No heartbeat from core client for 30 sec - exiting 18:09:04 (3696): No heartbeat from core client for 30 sec - exiting 18:09:05 (3696): No heartbeat from core client for 30 sec - exiting 18:09:07 (3696): No heartbeat from core client for 30 sec - exiting 18:09:08 (3696): No heartbeat from core client for 30 sec - exiting 18:09:09 (3696): No heartbeat from core client for 30 sec - exiting 18:09:10 (3696): No heartbeat from core client for 30 sec - exiting 18:09:11 (3696): No heartbeat from core client for 30 sec - exiting 18:09:12 (3696): No heartbeat from core client for 30 sec - exiting 18:09:13 (3696): No heartbeat from core client for 30 sec - exiting 18:09:14 (3696): No heartbeat from core client for 30 sec - exiting 18:09:15 (3696): No heartbeat from core client for 30 sec - exiting 18:09:16 (3696): No heartbeat from core client for 30 sec - exiting 18:09:17 (3696): No heartbeat from core client for 30 sec - exiting 18:09:18 (3696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:03 (4816): No heartbeat from core client for 30 sec - exiting 06:36:04 (4816): No heartbeat from core client for 30 sec - exiting 06:36:05 (4816): No heartbeat from core client for 30 sec - exiting 06:36:06 (4816): No heartbeat from core client for 30 sec - exiting 06:36:07 (4816): No heartbeat from core client for 30 sec - exiting 06:36:08 (4816): No heartbeat from core client for 30 sec - exiting 06:36:09 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:43 (4160): No heartbeat from core client for 30 sec - exiting 18:33:44 (4160): No heartbeat from core client for 30 sec - exiting 18:33:46 (4160): No heartbeat from core client for 30 sec - exiting 18:33:47 (4160): No heartbeat from core client for 30 sec - exiting 18:33:48 (4160): No heartbeat from core client for 30 sec - exiting 18:33:49 (4160): No heartbeat from core client for 30 sec - exiting 18:33:50 (4160): No heartbeat from core client for 30 sec - exiting 18:33:51 (4160): No heartbeat from core client for 30 sec - exiting 18:33:52 (4160): No heartbeat from core client for 30 sec - exiting 18:33:53 (4160): No heartbeat from core client for 30 sec - exiting 18:33:54 (4160): No heartbeat from core client for 30 sec - exiting 18:33:55 (4160): No heartbeat from core client for 30 sec - exiting 18:33:56 (4160): No heartbeat from core client for 30 sec - exiting 18:33:58 (4160): No heartbeat from core client for 30 sec - exiting 18:33:59 (4160): No heartbeat from core client for 30 sec - exiting 18:34:00 (4160): No heartbeat from core client for 30 sec - exiting 18:34:01 (4160): No heartbeat from core client for 30 sec - exiting 18:34:02 (4160): No heartbeat from core client for 30 sec - exiting 18:34:03 (4160): No heartbeat from core client for 30 sec - exiting 18:34:04 (4160): No heartbeat from core client for 30 sec - exiting 18:34:05 (4160): No heartbeat from core client for 30 sec - exiting 18:34:06 (4160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:07 (4160): No heartbeat from core client for 30 sec - exiting 18:42:20 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5252, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4880, iMonCtr=1 Model crash detected, will try to restart... 17:25:20 (4600): No heartbeat from core client for 30 sec - exiting 17:25:21 (4600): No heartbeat from core client for 30 sec - exiting 17:25:22 (4600): No heartbeat from core client for 30 sec - exiting 17:25:23 (4600): No heartbeat from core client for 30 sec - exiting 17:25:24 (4600): No heartbeat from core client for 30 sec - exiting 17:25:25 (4600): No heartbeat from core client for 30 sec - exiting 17:25:27 (4600): No heartbeat from core client for 30 sec - exiting 17:25:28 (4600): No heartbeat from core client for 30 sec - exiting 17:25:29 (4600): No heartbeat from core client for 30 sec - exiting 17:25:30 (4600): No heartbeat from core client for 30 sec - exiting 17:25:31 (4600): No heartbeat from core client for 30 sec - exiting 17:25:32 (4600): No heartbeat from core client for 30 sec - exiting 17:25:33 (4600): No heartbeat from core client for 30 sec - exiting 17:25:34 (4600): No heartbeat from core client for 30 sec - exiting 17:25:35 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2011 22:18:14	1036046	13540620	hadcm3n_yf8d_1900_40_007517471_0	103,680	137,833	1.3294
27 Nov 2011 17:18:04	1036046	13540620	hadcm3n_yf8d_1900_40_007517471_0	77,760	103,436	1.3302
26 Nov 2011 19:41:07	1036046	13540620	hadcm3n_yf8d_1900_40_007517471_0	51,840	68,997	1.3310
26 Nov 2011 01:02:39	1036046	13540620	hadcm3n_yf8d_1900_40_007517471_0	25,920	34,169	1.3182