Task 15491294

Name	hadcm3n_3ja1_1940_40_008262708_0
Workunit	8417832
Created	21 Dec 2012, 2:42:23 UTC
Sent	28 Dec 2012, 16:11:22 UTC
Report deadline	29 Mar 2013, 23:38:33 UTC
Received	2 Feb 2013, 4:16:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1056185
Run time	33 days 16 hours 33 min 18 sec
CPU time	23 days 18 hours 38 min 34 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	1.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 12:29:48 (1368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:28:45 (2176): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:27:48 (3568): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:27:50 (3568): No heartbeat from core client for 30 sec - exiting 23:27:51 (3568): No heartbeat from core client for 30 sec - exiting 21:23:19 (2960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:18 (2232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:14 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:12 (976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:30:41 (612): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:17:21 (2360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:57:19 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:57:20 (4156): No heartbeat from core client for 30 sec - exiting 20:16:11 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:15:06 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:00 (1256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:55 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:51 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:10:43 (2556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:36 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:34 (2672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:31 (728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:30 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:32 (3812): No heartbeat from core client for 30 sec - exiting 07:05:25 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:16 (2672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:03:11 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:07 (2400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:01:01 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:59:58 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:58:52 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:32 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:26 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:52:23 (960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:28 (2468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:30 (2468): No heartbeat from core client for 30 sec - exiting 08:51:31 (2468): No heartbeat from core client for 30 sec - exiting 08:51:33 (2468): No heartbeat from core client for 30 sec - exiting 14:50:13 (5432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:21 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:48 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:48 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:40 (5640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:34 (1356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:31 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:27 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:24 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:21 (608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:36:17 (5348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:13 (1328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:10 (604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:05 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:07 (5480): No heartbeat from core client for 30 sec - exiting 13:31:55 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:54 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:54 (3232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:45 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:27:41 (5168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:38 (1088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:15 (4536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:09 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:21:05 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:21:06 (5516): No heartbeat from core client for 30 sec - exiting 04:20:00 (1328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:16 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:20 (4172): No heartbeat from core client for 30 sec - exiting 09:51:22 (4172): No heartbeat from core client for 30 sec - exiting 09:51:27 (4172): No heartbeat from core client for 30 sec - exiting 09:51:29 (4172): No heartbeat from core client for 30 sec - exiting 09:51:30 (4172): No heartbeat from core client for 30 sec - exiting 10:18:56 (608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:17:53 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:43 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:41:41 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:34:57 (292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:54 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:52 (2204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:54 (3092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:46 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:47 (5252): No heartbeat from core client for 30 sec - exiting 14:29:45 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:46 (5548): No heartbeat from core client for 30 sec - exiting 20:28:43 (700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:52 (1932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:41 (2332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:33 (2116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:26 (5184): No heartbeat from core client for 30 sec - exiting 22:24:28 (5184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:30 (5184): No heartbeat from core client for 30 sec - exiting 22:24:31 (5184): No heartbeat from core client for 30 sec - exiting 22:24:32 (5184): No heartbeat from core client for 30 sec - exiting 22:24:34 (5184): No heartbeat from core client for 30 sec - exiting 22:24:35 (5184): No heartbeat from core client for 30 sec - exiting 22:24:36 (5184): No heartbeat from core client for 30 sec - exiting 04:23:19 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:19 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:08 (3316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:10 (3316): No heartbeat from core client for 30 sec - exiting 08:17:43 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:16:41 (2484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:42 (5244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 03:14:45 (548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:14:47 (548): No heartbeat from core client for 30 sec - exiting 09:13:30 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:31 (2240): No heartbeat from core client for 30 sec - exiting 16:12:21 (1120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:13 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:05 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:19 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:11 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:06:04 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:11 (4912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:04:02 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:53 (488): No heartbeat from core client for 30 sec - exiting 17:02:55 (488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:56 (488): No heartbeat from core client for 30 sec - exiting 17:02:57 (488): No heartbeat from core client for 30 sec - exiting 00:01:45 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:01:46 (4592): No heartbeat from core client for 30 sec - exiting 06:00:44 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:59:43 (3732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:59:44 (3732): No heartbeat from core client for 30 sec - exiting 18:58:33 (2336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:34 (1940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:33 (5156): No heartbeat from core client for 30 sec - exiting 06:56:34 (5156): No heartbeat from core client for 30 sec - exiting 06:56:36 (5156): No heartbeat from core client for 30 sec - exiting 06:56:37 (5156): No heartbeat from core client for 30 sec - exiting 06:56:38 (5156): No heartbeat from core client for 30 sec - exiting 06:56:39 (5156): No heartbeat from core client for 30 sec - exiting 06:56:40 (5156): No heartbeat from core client for 30 sec - exiting 06:56:41 (5156): No heartbeat from core client for 30 sec - exiting 06:56:42 (5156): No heartbeat from core client for 30 sec - exiting 06:56:44 (5156): No heartbeat from core client for 30 sec - exiting 06:56:45 (5156): No heartbeat from core client for 30 sec - exiting 06:56:46 (5156): No heartbeat from core client for 30 sec - exiting 06:56:47 (5156): No heartbeat from core client for 30 sec - exiting 06:56:48 (5156): No heartbeat from core client for 30 sec - exiting 06:56:49 (5156): No heartbeat from core client for 30 sec - exiting 06:56:51 (5156): No heartbeat from core client for 30 sec - exiting 06:56:52 (5156): No heartbeat from core client for 30 sec - exiting 06:56:53 (5156): No heartbeat from core client for 30 sec - exiting 06:56:54 (5156): No heartbeat from core client for 30 sec - exiting 06:56:55 (5156): No heartbeat from core client for 30 sec - exiting 06:56:56 (5156): No heartbeat from core client for 30 sec - exiting 06:56:57 (5156): No heartbeat from core client for 30 sec - exiting 06:56:59 (5156): No heartbeat from core client for 30 sec - exiting 06:57:00 (5156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3ja1ko.pjf4c10 Error converting file to netcdf: dataout/3ja1ko.pif4c10 Error converting file to netcdf: dataout/3ja1ko.pff4c10 Error converting file to netcdf: dataout/3ja1ka.phf4c10 Error converting file to netcdf: dataout/3ja1ka.pgf4c10 Error converting file to netcdf: dataout/3ja1ka.pef4c10 Error converting file to netcdf: dataout/3ja1ka.pdf4c10 13:55:26 (2312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:27 (2312): No heartbeat from core client for 30 sec - exiting 20:54:20 (5668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:22 (5668): No heartbeat from core client for 30 sec - exiting 11:50:50 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:51 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:53 (1688): No heartbeat from core client for 30 sec - exiting 23:48:50 (780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:49 (1608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:51 (1608): No heartbeat from core client for 30 sec - exiting 05:47:52 (1608): No heartbeat from core client for 30 sec - exiting 11:46:54 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:46:55 (6108): No heartbeat from core client for 30 sec - exiting 11:46:57 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:42:52 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:42:54 (3604): No heartbeat from core client for 30 sec - exiting 14:42:55 (3604): No heartbeat from core client for 30 sec - exiting 14:42:57 (3604): No heartbeat from core client for 30 sec - exiting 14:42:58 (3604): No heartbeat from core client for 30 sec - exiting 19:29:16 (1632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:10 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:48 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:35 (2808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:28 (1200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:29 (1200): No heartbeat from core client for 30 sec - exiting 13:25:30 (1200): No heartbeat from core client for 30 sec - exiting 13:25:32 (1200): No heartbeat from core client for 30 sec - exiting 20:24:03 (3996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:44 (2108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:44 (3564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:44 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:14 (692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:44 (4596): No heartbeat from core client for 30 sec - exiting 23:21:47 (4596): No heartbeat from core client for 30 sec - exiting 23:21:48 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:50 (4596): No heartbeat from core client for 30 sec - exiting 23:21:51 (4596): No heartbeat from core client for 30 sec - exiting 23:21:52 (4596): No heartbeat from core client for 30 sec - exiting 23:21:53 (4596): No heartbeat from core client for 30 sec - exiting 23:21:55 (4596): No heartbeat from core client for 30 sec - exiting 23:21:56 (4596): No heartbeat from core client for 30 sec - exiting 23:21:57 (4596): No heartbeat from core client for 30 sec - exiting 23:21:58 (4596): No heartbeat from core client for 30 sec - exiting 02:17:59 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:47 (908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:48 (908): No heartbeat from core client for 30 sec - exiting 08:16:49 (908): No heartbeat from core client for 30 sec - exiting 08:16:51 (908): No heartbeat from core client for 30 sec - exiting 14:15:33 (2124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:17 (6068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:06 (488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:49 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:51 (4108): No heartbeat from core client for 30 sec - exiting 08:11:52 (4108): No heartbeat from core client for 30 sec - exiting 14:10:35 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:52 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:00 (2192): No heartbeat from core client for 30 sec - exiting 15:03:01 (2192): No heartbeat from core client for 30 sec - exiting 15:03:03 (2192): No heartbeat from core client for 30 sec - exiting 15:03:04 (2192): No heartbeat from core client for 30 sec - exiting 15:03:05 (2192): No heartbeat from core client for 30 sec - exiting 15:03:06 (2192): No heartbeat from core client for 30 sec - exiting 15:03:07 (2192): No heartbeat from core client for 30 sec - exiting 15:03:08 (2192): No heartbeat from core client for 30 sec - exiting 15:03:09 (2192): No heartbeat from core client for 30 sec - exiting 15:03:10 (2192): No heartbeat from core client for 30 sec - exiting 15:03:12 (2192): No heartbeat from core client for 30 sec - exiting 15:03:13 (2192): No heartbeat from core client for 30 sec - exiting 15:12:19 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:21 (4664): No heartbeat from core client for 30 sec - exiting 15:12:22 (4664): No heartbeat from core client for 30 sec - exiting 15:12:23 (4664): No heartbeat from core client for 30 sec - exiting 20:09:23 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:09:24 (1352): No heartbeat from core client for 30 sec - exiting 02:08:04 (1516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:34 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:03 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:29 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:31 (1116): No heartbeat from core client for 30 sec - exiting 14:05:32 (1116): No heartbeat from core client for 30 sec - exiting 20:04:54 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:04:56 (2508): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3620, iMonCtr=1 Model crash detected, will try to restart... 21:04:06 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=808, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=808, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Jan 2013 13:55:24	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	414,720	1,985,067	4.7865
28 Jan 2013 15:30:34	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	388,800	1,832,113	4.7122
25 Jan 2013 13:50:21	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	362,880	1,682,238	4.6358
22 Jan 2013 23:01:19	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	336,960	1,526,347	4.5298
20 Jan 2013 14:05:35	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	311,040	1,369,773	4.4038
17 Jan 2013 03:13:26	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	285,120	1,213,782	4.2571
14 Jan 2013 13:35:25	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	259,200	1,077,036	4.1552
13 Jan 2013 03:48:41	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	233,280	971,178	4.1631
11 Jan 2013 06:58:59	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	207,360	866,008	4.1764
09 Jan 2013 19:34:07	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	181,440	760,377	4.1908
08 Jan 2013 08:42:02	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	155,520	654,906	4.2111
06 Jan 2013 22:08:24	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	129,600	549,504	4.2400
05 Jan 2013 18:33:00	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	103,680	444,450	4.2867
03 Jan 2013 17:31:31	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	77,760	339,333	4.3639
02 Jan 2013 06:34:37	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	51,840	233,802	4.5101
31 Dec 2012 10:58:54	1056185	15491294	hadcm3n_3ja1_1940_40_008262708_0	25,920	105,062	4.0533