Task 14841426

Name	hadcm3n_yb7d_1980_40_008026134_2
Workunit	8181248
Created	25 Jun 2012, 4:42:11 UTC
Sent	25 Jun 2012, 4:42:34 UTC
Report deadline	24 Sep 2012, 12:09:45 UTC
Received	6 Jul 2012, 7:03:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1099619
Run time	3 days 21 hours 5 min 49 sec
CPU time	3 days 10 hours 38 min 42 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	3.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:21:59 (10340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:00 (10340): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:36:24 (6924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:25 (6924): No heartbeat from core client for 30 sec - exiting 15:36:26 (6924): No heartbeat from core client for 30 sec - exiting 15:36:27 (6924): No heartbeat from core client for 30 sec - exiting 15:36:28 (6924): No heartbeat from core client for 30 sec - exiting 15:36:29 (6924): No heartbeat from core client for 30 sec - exiting 15:36:30 (6924): No heartbeat from core client for 30 sec - exiting 15:36:31 (6924): No heartbeat from core client for 30 sec - exiting 15:36:32 (6924): No heartbeat from core client for 30 sec - exiting 15:36:33 (6924): No heartbeat from core client for 30 sec - exiting 15:36:34 (6924): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on yb7dko.dai4ah0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:25:59 (6428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:26:00 (6428): No heartbeat from core client for 30 sec - exiting 17:26:01 (6428): No heartbeat from core client for 30 sec - exiting 17:26:02 (6428): No heartbeat from core client for 30 sec - exiting 17:26:03 (6428): No heartbeat from core client for 30 sec - exiting 17:26:04 (6428): No heartbeat from core client for 30 sec - exiting 17:26:05 (6428): No heartbeat from core client for 30 sec - exiting 17:26:06 (6428): No heartbeat from core client for 30 sec - exiting 17:26:07 (6428): No heartbeat from core client for 30 sec - exiting 17:26:08 (6428): No heartbeat from core client for 30 sec - exiting 17:26:09 (6428): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:37:26 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:37:27 (3600): No heartbeat from core client for 30 sec - exiting 17:37:28 (3600): No heartbeat from core client for 30 sec - exiting 17:37:29 (3600): No heartbeat from core client for 30 sec - exiting 17:37:30 (3600): No heartbeat from core client for 30 sec - exiting 17:37:31 (3600): No heartbeat from core client for 30 sec - exiting 17:37:32 (3600): No heartbeat from core client for 30 sec - exiting 17:37:33 (3600): No heartbeat from core client for 30 sec - exiting 17:37:34 (3600): No heartbeat from core client for 30 sec - exiting 17:37:35 (3600): No heartbeat from core client for 30 sec - exiting 17:37:36 (3600): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:57:59 (8188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:00 (8188): No heartbeat from core client for 30 sec - exiting 17:58:01 (8188): No heartbeat from core client for 30 sec - exiting 17:58:02 (8188): No heartbeat from core client for 30 sec - exiting 17:58:03 (8188): No heartbeat from core client for 30 sec - exiting 17:58:04 (8188): No heartbeat from core client for 30 sec - exiting 17:58:05 (8188): No heartbeat from core client for 30 sec - exiting 17:58:06 (8188): No heartbeat from core client for 30 sec - exiting 17:58:07 (8188): No heartbeat from core client for 30 sec - exiting 17:58:08 (8188): No heartbeat from core client for 30 sec - exiting 17:58:09 (8188): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:13:00 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:01 (4048): No heartbeat from core client for 30 sec - exiting 18:13:02 (4048): No heartbeat from core client for 30 sec - exiting 18:13:03 (4048): No heartbeat from core client for 30 sec - exiting 18:13:04 (4048): No heartbeat from core client for 30 sec - exiting 18:13:05 (4048): No heartbeat from core client for 30 sec - exiting 18:13:06 (4048): No heartbeat from core client for 30 sec - exiting 18:13:07 (4048): No heartbeat from core client for 30 sec - exiting 18:13:08 (4048): No heartbeat from core client for 30 sec - exiting 18:13:09 (4048): No heartbeat from core client for 30 sec - exiting 18:13:10 (4048): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:33:01 (6600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:02 (6600): No heartbeat from core client for 30 sec - exiting 18:33:03 (6600): No heartbeat from core client for 30 sec - exiting 18:33:04 (6600): No heartbeat from core client for 30 sec - exiting 18:33:05 (6600): No heartbeat from core client for 30 sec - exiting 18:33:06 (6600): No heartbeat from core client for 30 sec - exiting 18:33:07 (6600): No heartbeat from core client for 30 sec - exiting 18:33:08 (6600): No heartbeat from core client for 30 sec - exiting 18:33:09 (6600): No heartbeat from core client for 30 sec - exiting 18:33:10 (6600): No heartbeat from core client for 30 sec - exiting 18:33:11 (6600): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:23:02 (7584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:03 (7584): No heartbeat from core client for 30 sec - exiting 21:23:04 (7584): No heartbeat from core client for 30 sec - exiting 21:23:05 (7584): No heartbeat from core client for 30 sec - exiting 21:23:06 (7584): No heartbeat from core client for 30 sec - exiting 21:23:07 (7584): No heartbeat from core client for 30 sec - exiting 21:23:08 (7584): No heartbeat from core client for 30 sec - exiting 21:23:09 (7584): No heartbeat from core client for 30 sec - exiting 21:23:10 (7584): No heartbeat from core client for 30 sec - exiting 21:23:11 (7584): No heartbeat from core client for 30 sec - exiting 21:23:12 (7584): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:20:04 (10112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:05 (10112): No heartbeat from core client for 30 sec - exiting 23:20:06 (10112): No heartbeat from core client for 30 sec - exiting 23:20:07 (10112): No heartbeat from core client for 30 sec - exiting 23:20:08 (10112): No heartbeat from core client for 30 sec - exiting 23:20:09 (10112): No heartbeat from core client for 30 sec - exiting 23:20:10 (10112): No heartbeat from core client for 30 sec - exiting 23:20:11 (10112): No heartbeat from core client for 30 sec - exiting 23:20:12 (10112): No heartbeat from core client for 30 sec - exiting 23:20:13 (10112): No heartbeat from core client for 30 sec - exiting 23:20:14 (10112): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:13:02 (3472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:03 (3472): No heartbeat from core client for 30 sec - exiting 07:13:04 (3472): No heartbeat from core client for 30 sec - exiting 07:13:05 (3472): No heartbeat from core client for 30 sec - exiting 07:13:06 (3472): No heartbeat from core client for 30 sec - exiting 07:13:07 (3472): No heartbeat from core client for 30 sec - exiting 07:13:08 (3472): No heartbeat from core client for 30 sec - exiting 07:13:09 (3472): No heartbeat from core client for 30 sec - exiting 07:13:10 (3472): No heartbeat from core client for 30 sec - exiting 07:13:11 (3472): No heartbeat from core client for 30 sec - exiting 07:13:12 (3472): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:14:36 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:35:40 (10056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:35:41 (10056): No heartbeat from core client for 30 sec - exiting 00:35:42 (10056): No heartbeat from core client for 30 sec - exiting 00:35:43 (10056): No heartbeat from core client for 30 sec - exiting 00:35:44 (10056): No heartbeat from core client for 30 sec - exiting 00:35:45 (10056): No heartbeat from core client for 30 sec - exiting 00:35:46 (10056): No heartbeat from core client for 30 sec - exiting 00:35:47 (10056): No heartbeat from core client for 30 sec - exiting 00:35:48 (10056): No heartbeat from core client for 30 sec - exiting 00:35:49 (10056): No heartbeat from core client for 30 sec - exiting 00:35:50 (10056): No heartbeat from core client for 30 sec - exiting 01:11:57 (11240): No heartbeat from core client for 30 sec - exiting 01:11:58 (11240): No heartbeat from core client for 30 sec - exiting 01:11:59 (11240): No heartbeat from core client for 30 sec - exiting 01:12:00 (11240): No heartbeat from core client for 30 sec - exiting 01:12:01 (11240): No heartbeat from core client for 30 sec - exiting 01:12:02 (11240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:12:50 (6328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:51 (6328): No heartbeat from core client for 30 sec - exiting 21:12:52 (6328): No heartbeat from core client for 30 sec - exiting 21:12:53 (6328): No heartbeat from core client for 30 sec - exiting 21:12:54 (6328): No heartbeat from core client for 30 sec - exiting 21:12:55 (6328): No heartbeat from core client for 30 sec - exiting 21:12:56 (6328): No heartbeat from core client for 30 sec - exiting 21:12:57 (6328): No heartbeat from core client for 30 sec - exiting 21:12:58 (6328): No heartbeat from core client for 30 sec - exiting 21:12:59 (6328): No heartbeat from core client for 30 sec - exiting 21:13:00 (6328): No heartbeat from core client for 30 sec - exiting 21:29:51 (11596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:52 (11596): No heartbeat from core client for 30 sec - exiting 21:29:53 (11596): No heartbeat from core client for 30 sec - exiting 21:29:54 (11596): No heartbeat from core client for 30 sec - exiting 21:29:55 (11596): No heartbeat from core client for 30 sec - exiting 21:29:56 (11596): No heartbeat from core client for 30 sec - exiting 21:29:57 (11596): No heartbeat from core client for 30 sec - exiting 21:29:58 (11596): No heartbeat from core client for 30 sec - exiting 21:29:59 (11596): No heartbeat from core client for 30 sec - exiting 21:30:00 (11596): No heartbeat from core client for 30 sec - exiting 21:30:01 (11596): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 21:41:15 (11916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:41:16 (11916): No heartbeat from core client for 30 sec - exiting 21:41:17 (11916): No heartbeat from core client for 30 sec - exiting 21:41:18 (11916): No heartbeat from core client for 30 sec - exiting 21:41:19 (11916): No heartbeat from core client for 30 sec - exiting 21:41:20 (11916): No heartbeat from core client for 30 sec - exiting 21:41:21 (11916): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:01:52 (9676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:06 (10188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:07 (10188): No heartbeat from core client for 30 sec - exiting 22:11:08 (10188): No heartbeat from core client for 30 sec - exiting 22:11:09 (10188): No heartbeat from core client for 30 sec - exiting 22:11:10 (10188): No heartbeat from core client for 30 sec - exiting 22:11:11 (10188): No heartbeat from core client for 30 sec - exiting 22:11:12 (10188): No heartbeat from core client for 30 sec - exiting 22:11:13 (10188): No heartbeat from core client for 30 sec - exiting 22:11:14 (10188): No heartbeat from core client for 30 sec - exiting 22:11:15 (10188): No heartbeat from core client for 30 sec - exiting 22:11:16 (10188): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11380, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2012 15:18:10	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	155,520	256,446	1.6490
03 Jul 2012 13:18:54	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	129,600	214,218	1.6529
02 Jul 2012 22:11:04	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	103,680	173,867	1.6770
26 Jun 2012 20:46:01	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	77,760	131,983	1.6973
26 Jun 2012 08:45:35	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	51,840	92,870	1.7915
25 Jun 2012 18:45:36	1099619	14841426	hadcm3n_yb7d_1980_40_008026134_2	25,920	46,785	1.8050