Task 16590116

Name	hadcm3n_8c80_1980_40_008725259_0
Workunit	8871237
Created	23 Apr 2014, 13:25:45 UTC
Sent	1 May 2014, 8:33:40 UTC
Report deadline	31 Jul 2014, 16:00:51 UTC
Received	8 Jun 2014, 12:37:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1286679
Run time	3 days 16 hours 49 min 56 sec
CPU time	3 days 0 hours 51 min 50 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.51 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:32:31 (13128): No heartbeat from core client for 30 sec - exiting 11:32:53 (13128): No heartbeat from core client for 30 sec - exiting 11:32:54 (13128): No heartbeat from core client for 30 sec - exiting 11:32:55 (13128): No heartbeat from core client for 30 sec - exiting 11:32:56 (13128): No heartbeat from core client for 30 sec - exiting 11:32:57 (13128): No heartbeat from core client for 30 sec - exiting 11:32:58 (13128): No heartbeat from core client for 30 sec - exiting 11:32:59 (13128): No heartbeat from core client for 30 sec - exiting 11:33:00 (13128): No heartbeat from core client for 30 sec - exiting 11:33:01 (13128): No heartbeat from core client for 30 sec - exiting 11:33:02 (13128): No heartbeat from core client for 30 sec - exiting 11:33:03 (13128): No heartbeat from core client for 30 sec - exiting 11:33:04 (13128): No heartbeat from core client for 30 sec - exiting 11:33:05 (13128): No heartbeat from core client for 30 sec - exiting 11:33:06 (13128): No heartbeat from core client for 30 sec - exiting 11:33:07 (13128): No heartbeat from core client for 30 sec - exiting 11:33:08 (13128): No heartbeat from core client for 30 sec - exiting 11:33:09 (13128): No heartbeat from core client for 30 sec - exiting 11:33:10 (13128): No heartbeat from core client for 30 sec - exiting 11:33:11 (13128): No heartbeat from core client for 30 sec - exiting 11:33:12 (13128): No heartbeat from core client for 30 sec - exiting 11:33:13 (13128): No heartbeat from core client for 30 sec - exiting 11:33:14 (13128): No heartbeat from core client for 30 sec - exiting 11:33:15 (13128): No heartbeat from core client for 30 sec - exiting 11:33:16 (13128): No heartbeat from core client for 30 sec - exiting 11:33:17 (13128): No heartbeat from core client for 30 sec - exiting 11:33:18 (13128): No heartbeat from core client for 30 sec - exiting 11:33:19 (13128): No heartbeat from core client for 30 sec - exiting 11:33:20 (13128): No heartbeat from core client for 30 sec - exiting 11:33:21 (13128): No heartbeat from core client for 30 sec - exiting 11:33:22 (13128): No heartbeat from core client for 30 sec - exiting 11:33:23 (13128): No heartbeat from core client for 30 sec - exiting 11:33:24 (13128): No heartbeat from core client for 30 sec - exiting 11:33:25 (13128): No heartbeat from core client for 30 sec - exiting 11:33:26 (13128): No heartbeat from core client for 30 sec - exiting 11:33:27 (13128): No heartbeat from core client for 30 sec - exiting 11:33:28 (13128): No heartbeat from core client for 30 sec - exiting 11:33:29 (13128): No heartbeat from core client for 30 sec - exiting 11:33:30 (13128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:49:50 (5896): No heartbeat from core client for 30 sec - exiting 16:49:51 (5896): No heartbeat from core client for 30 sec - exiting 16:49:52 (5896): No heartbeat from core client for 30 sec - exiting 16:49:53 (5896): No heartbeat from core client for 30 sec - exiting 16:49:54 (5896): No heartbeat from core client for 30 sec - exiting 16:49:55 (5896): No heartbeat from core client for 30 sec - exiting 16:49:56 (5896): No heartbeat from core client for 30 sec - exiting 16:49:57 (5896): No heartbeat from core client for 30 sec - exiting 16:49:58 (5896): No heartbeat from core client for 30 sec - exiting 16:49:59 (5896): No heartbeat from core client for 30 sec - exiting 16:50:00 (5896): No heartbeat from core client for 30 sec - exiting 16:50:01 (5896): No heartbeat from core client for 30 sec - exiting 16:50:02 (5896): No heartbeat from core client for 30 sec - exiting 16:50:03 (5896): No heartbeat from core client for 30 sec - exiting 16:50:04 (5896): No heartbeat from core client for 30 sec - exiting 16:50:05 (5896): No heartbeat from core client for 30 sec - exiting 16:50:06 (5896): No heartbeat from core client for 30 sec - exiting 16:50:07 (5896): No heartbeat from core client for 30 sec - exiting 16:50:08 (5896): No heartbeat from core client for 30 sec - exiting 16:50:09 (5896): No heartbeat from core client for 30 sec - exiting 16:50:10 (5896): No heartbeat from core client for 30 sec - exiting 16:50:11 (5896): No heartbeat from core client for 30 sec - exiting 16:50:12 (5896): No heartbeat from core client for 30 sec - exiting 16:50:13 (5896): No heartbeat from core client for 30 sec - exiting 16:50:14 (5896): No heartbeat from core client for 30 sec - exiting 16:50:15 (5896): No heartbeat from core client for 30 sec - exiting 16:50:16 (5896): No heartbeat from core client for 30 sec - exiting 16:50:17 (5896): No heartbeat from core client for 30 sec - exiting 16:50:18 (5896): No heartbeat from core client for 30 sec - exiting 16:50:19 (5896): No heartbeat from core client for 30 sec - exiting 16:50:20 (5896): No heartbeat from core client for 30 sec - exiting 16:50:21 (5896): No heartbeat from core client for 30 sec - exiting 16:50:22 (5896): No heartbeat from core client for 30 sec - exiting 16:50:23 (5896): No heartbeat from core client for 30 sec - exiting 16:50:24 (5896): No heartbeat from core client for 30 sec - exiting 16:50:25 (5896): No heartbeat from core client for 30 sec - exiting 16:50:26 (5896): No heartbeat from core client for 30 sec - exiting 16:50:27 (5896): No heartbeat from core client for 30 sec - exiting 16:50:28 (5896): No heartbeat from core client for 30 sec - exiting 16:50:29 (5896): No heartbeat from core client for 30 sec - exiting 16:50:30 (5896): No heartbeat from core client for 30 sec - exiting 16:50:31 (5896): No heartbeat from core client for 30 sec - exiting 16:50:32 (5896): No heartbeat from core client for 30 sec - exiting 16:50:33 (5896): No heartbeat from core client for 30 sec - exiting 16:50:34 (5896): No heartbeat from core client for 30 sec - exiting 16:50:35 (5896): No heartbeat from core client for 30 sec - exiting 16:50:36 (5896): No heartbeat from core client for 30 sec - exiting 16:50:37 (5896): No heartbeat from core client for 30 sec - exiting 16:50:38 (5896): No heartbeat from core client for 30 sec - exiting 16:50:39 (5896): No heartbeat from core client for 30 sec - exiting 16:50:40 (5896): No heartbeat from core client for 30 sec - exiting 16:50:41 (5896): No heartbeat from core client for 30 sec - exiting 16:50:42 (5896): No heartbeat from core client for 30 sec - exiting 16:50:43 (5896): No heartbeat from core client for 30 sec - exiting 16:50:44 (5896): No heartbeat from core client for 30 sec - exiting 16:50:45 (5896): No heartbeat from core client for 30 sec - exiting 16:50:46 (5896): No heartbeat from core client for 30 sec - exiting 16:50:47 (5896): No heartbeat from core client for 30 sec - exiting 16:50:48 (5896): No heartbeat from core client for 30 sec - exiting 16:50:49 (5896): No heartbeat from core client for 30 sec - exiting 16:50:50 (5896): No heartbeat from core client for 30 sec - exiting 16:50:51 (5896): No heartbeat from core client for 30 sec - exiting 16:50:52 (5896): No heartbeat from core client for 30 sec - exiting 16:50:53 (5896): No heartbeat from core client for 30 sec - exiting 16:50:54 (5896): No heartbeat from core client for 30 sec - exiting 16:50:55 (5896): No heartbeat from core client for 30 sec - exiting 16:50:56 (5896): No heartbeat from core client for 30 sec - exiting 16:50:57 (5896): No heartbeat from core client for 30 sec - exiting 16:50:58 (5896): No heartbeat from core client for 30 sec - exiting 16:50:59 (5896): No heartbeat from core client for 30 sec - exiting 16:51:00 (5896): No heartbeat from core client for 30 sec - exiting 16:51:01 (5896): No heartbeat from core client for 30 sec - exiting 16:51:02 (5896): No heartbeat from core client for 30 sec - exiting 16:51:03 (5896): No heartbeat from core client for 30 sec - exiting 16:51:04 (5896): No heartbeat from core client for 30 sec - exiting 16:51:05 (5896): No heartbeat from core client for 30 sec - exiting 16:51:06 (5896): No heartbeat from core client for 30 sec - exiting 16:51:07 (5896): No heartbeat from core client for 30 sec - exiting 16:51:08 (5896): No heartbeat from core client for 30 sec - exiting 16:51:09 (5896): No heartbeat from core client for 30 sec - exiting 16:51:10 (5896): No heartbeat from core client for 30 sec - exiting 16:51:11 (5896): No heartbeat from core client for 30 sec - exiting 16:51:12 (5896): No heartbeat from core client for 30 sec - exiting 16:51:13 (5896): No heartbeat from core client for 30 sec - exiting 16:51:14 (5896): No heartbeat from core client for 30 sec - exiting 16:51:15 (5896): No heartbeat from core client for 30 sec - exiting 16:51:16 (5896): No heartbeat from core client for 30 sec - exiting 16:51:17 (5896): No heartbeat from core client for 30 sec - exiting 16:51:18 (5896): No heartbeat from core client for 30 sec - exiting 16:51:19 (5896): No heartbeat from core client for 30 sec - exiting 16:51:20 (5896): No heartbeat from core client for 30 sec - exiting 16:51:21 (5896): No heartbeat from core client for 30 sec - exiting 16:51:22 (5896): No heartbeat from core client for 30 sec - exiting 16:51:23 (5896): No heartbeat from core client for 30 sec - exiting 16:51:24 (5896): No heartbeat from core client for 30 sec - exiting 16:51:25 (5896): No heartbeat from core client for 30 sec - exiting 16:51:26 (5896): No heartbeat from core client for 30 sec - exiting 16:51:27 (5896): No heartbeat from core client for 30 sec - exiting 16:51:28 (5896): No heartbeat from core client for 30 sec - exiting 16:51:29 (5896): No heartbeat from core client for 30 sec - exiting 16:51:30 (5896): No heartbeat from core client for 30 sec - exiting 16:51:31 (5896): No heartbeat from core client for 30 sec - exiting 16:51:32 (5896): No heartbeat from core client for 30 sec - exiting 16:51:33 (5896): No heartbeat from core client for 30 sec - exiting 16:51:34 (5896): No heartbeat from core client for 30 sec - exiting 16:51:35 (5896): No heartbeat from core client for 30 sec - exiting 16:51:36 (5896): No heartbeat from core client for 30 sec - exiting 16:51:37 (5896): No heartbeat from core client for 30 sec - exiting 16:51:38 (5896): No heartbeat from core client for 30 sec - exiting 16:51:39 (5896): No heartbeat from core client for 30 sec - exiting 16:51:40 (5896): No heartbeat from core client for 30 sec - exiting 16:51:41 (5896): No heartbeat from core client for 30 sec - exiting 16:51:42 (5896): No heartbeat from core client for 30 sec - exiting 16:51:43 (5896): No heartbeat from core client for 30 sec - exiting 16:51:44 (5896): No heartbeat from core client for 30 sec - exiting 16:51:45 (5896): No heartbeat from core client for 30 sec - exiting 16:51:46 (5896): No heartbeat from core client for 30 sec - exiting 16:51:47 (5896): No heartbeat from core client for 30 sec - exiting 16:51:48 (5896): No heartbeat from core client for 30 sec - exiting 16:51:49 (5896): No heartbeat from core client for 30 sec - exiting 16:51:50 (5896): No heartbeat from core client for 30 sec - exiting 16:51:51 (5896): No heartbeat from core client for 30 sec - exiting 16:51:52 (5896): No heartbeat from core client for 30 sec - exiting 16:51:53 (5896): No heartbeat from core client for 30 sec - exiting 16:51:54 (5896): No heartbeat from core client for 30 sec - exiting 16:51:55 (5896): No heartbeat from core client for 30 sec - exiting 16:51:56 (5896): No heartbeat from core client for 30 sec - exiting 16:51:57 (5896): No heartbeat from core client for 30 sec - exiting 16:51:58 (5896): No heartbeat from core client for 30 sec - exiting 16:51:59 (5896): No heartbeat from core client for 30 sec - exiting 16:52:00 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 8c80ko.dai16p0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=976, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 21:23:08 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:04:16 (4336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:56 (6784): No heartbeat from core client for 30 sec - exiting 11:00:59 (6784): No heartbeat from core client for 30 sec - exiting 11:01:00 (6784): No heartbeat from core client for 30 sec - exiting 11:01:01 (6784): No heartbeat from core client for 30 sec - exiting 11:01:02 (6784): No heartbeat from core client for 30 sec - exiting 11:01:03 (6784): No heartbeat from core client for 30 sec - exiting 11:01:04 (6784): No heartbeat from core client for 30 sec - exiting 11:01:05 (6784): No heartbeat from core client for 30 sec - exiting 11:01:06 (6784): No heartbeat from core client for 30 sec - exiting 11:01:07 (6784): No heartbeat from core client for 30 sec - exiting 11:01:08 (6784): No heartbeat from core client for 30 sec - exiting 11:01:09 (6784): No heartbeat from core client for 30 sec - exiting 11:01:10 (6784): No heartbeat from core client for 30 sec - exiting 11:01:11 (6784): No heartbeat from core client for 30 sec - exiting 11:01:12 (6784): No heartbeat from core client for 30 sec - exiting 11:01:13 (6784): No heartbeat from core client for 30 sec - exiting 11:01:14 (6784): No heartbeat from core client for 30 sec - exiting 11:01:15 (6784): No heartbeat from core client for 30 sec - exiting 11:01:16 (6784): No heartbeat from core client for 30 sec - exiting 11:01:17 (6784): No heartbeat from core client for 30 sec - exiting 11:01:18 (6784): No heartbeat from core client for 30 sec - exiting 11:01:19 (6784): No heartbeat from core client for 30 sec - exiting 11:01:20 (6784): No heartbeat from core client for 30 sec - exiting 11:01:21 (6784): No heartbeat from core client for 30 sec - exiting 11:01:22 (6784): No heartbeat from core client for 30 sec - exiting 11:01:23 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:30:55 (17896): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:40:00 (37232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:30 (36900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:25:42 (39396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:26:51 (40480): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 17:41:10 (42688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:41:11 (42688): No heartbeat from core client for 30 sec - exiting 17:41:12 (42688): No heartbeat from core client for 30 sec - exiting 17:41:13 (42688): No heartbeat from core client for 30 sec - exiting 17:41:14 (42688): No heartbeat from core client for 30 sec - exiting 17:41:15 (42688): No heartbeat from core client for 30 sec - exiting 17:41:16 (42688): No heartbeat from core client for 30 sec - exiting 17:41:17 (42688): No heartbeat from core client for 30 sec - exiting 17:41:18 (42688): No heartbeat from core client for 30 sec - exiting 17:41:19 (42688): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:01:31 (5396): No heartbeat from core client for 30 sec - exiting 08:01:32 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:02 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3432, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5744, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5744, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... 13:44:16 (5408): No heartbeat from core client for 30 sec - exiting 13:44:17 (5408): No heartbeat from core client for 30 sec - exiting 13:44:18 (5408): No heartbeat from core client for 30 sec - exiting 13:44:19 (5408): No heartbeat from core client for 30 sec - exiting 13:44:20 (5408): No heartbeat from core client for 30 sec - exiting 13:44:21 (5408): No heartbeat from core client for 30 sec - exiting 13:44:22 (5408): No heartbeat from core client for 30 sec - exiting 13:44:23 (5408): No heartbeat from core client for 30 sec - exiting 13:44:24 (5408): No heartbeat from core client for 30 sec - exiting 13:44:25 (5408): No heartbeat from core client for 30 sec - exiting 13:44:26 (5408): No heartbeat from core client for 30 sec - exiting 13:44:27 (5408): No heartbeat from core client for 30 sec - exiting 13:44:28 (5408): No heartbeat from core client for 30 sec - exiting 13:44:29 (5408): No heartbeat from core client for 30 sec - exiting 13:44:30 (5408): No heartbeat from core client for 30 sec - exiting 13:44:31 (5408): No heartbeat from core client for 30 sec - exiting 13:44:32 (5408): No heartbeat from core client for 30 sec - exiting 13:44:33 (5408): No heartbeat from core client for 30 sec - exiting 13:44:34 (5408): No heartbeat from core client for 30 sec - exiting 13:44:35 (5408): No heartbeat from core client for 30 sec - exiting 13:44:36 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6264, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6264, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jun 2014 09:01:56	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	155,520	244,910	1.5748
10 Jun 2014 09:01:56	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	129,600	210,966	1.6278
10 Jun 2014 09:01:56	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	103,680	174,438	1.6825
10 Jun 2014 09:01:56	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	77,760	136,363	1.7536
12 May 2014 12:08:53	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	51,840	90,513	1.7460
10 May 2014 06:32:53	1286679	16590116	hadcm3n_8c80_1980_40_008725259_0	25,920	43,102	1.6629