Task 15650266

Name	hadcm3n_3j9y_1980_40_008319498_1
Workunit	8470633
Created	5 Mar 2013, 18:50:48 UTC
Sent	5 Mar 2013, 18:50:54 UTC
Report deadline	5 Jun 2013, 2:18:05 UTC
Received	8 Jul 2013, 20:50:18 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1241496
Run time	60 days 13 hours 35 min 59 sec
CPU time	46 days 22 hours 11 min 7 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	1.94 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <stderr_txt> 01:30:21 (3396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:51:55 (2724): No heartbeat from core client for 30 sec - exiting 01:51:56 (2724): No heartbeat from core client for 30 sec - exiting 01:51:57 (2724): No heartbeat from core client for 30 sec - exiting 01:51:58 (2724): No heartbeat from core client for 30 sec - exiting 01:52:00 (2724): No heartbeat from core client for 30 sec - exiting 01:52:01 (2724): No heartbeat from core client for 30 sec - exiting 01:52:02 (2724): No heartbeat from core client for 30 sec - exiting 01:52:03 (2724): No heartbeat from core client for 30 sec - exiting 01:52:04 (2724): No heartbeat from core client for 30 sec - exiting 01:52:05 (2724): No heartbeat from core client for 30 sec - exiting 01:52:06 (2724): No heartbeat from core client for 30 sec - exiting 01:52:07 (2724): No heartbeat from core client for 30 sec - exiting 01:52:09 (2724): No heartbeat from core client for 30 sec - exiting 01:52:10 (2724): No heartbeat from core client for 30 sec - exiting 01:52:12 (2724): No heartbeat from core client for 30 sec - exiting 01:52:13 (2724): No heartbeat from core client for 30 sec - exiting 01:52:14 (2724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3j9yko.pji1c10 Error converting file to netcdf: dataout/3j9yko.pii1c10 Error converting file to netcdf: dataout/3j9yko.pfi1c10 Error converting file to netcdf: dataout/3j9yka.phi1c10 Error converting file to netcdf: dataout/3j9yka.pgi1c10 Error converting file to netcdf: dataout/3j9yka.pei1c10 Error converting file to netcdf: dataout/3j9yka.pdi1c10 02:01:03 (3596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:32 (168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:38 (1776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:48 (152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:28 (580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:29 (580): No heartbeat from core client for 30 sec - exiting 09:14:30 (580): No heartbeat from core client for 30 sec - exiting 09:14:31 (580): No heartbeat from core client for 30 sec - exiting 09:14:32 (580): No heartbeat from core client for 30 sec - exiting 09:14:33 (580): No heartbeat from core client for 30 sec - exiting 09:14:34 (580): No heartbeat from core client for 30 sec - exiting 09:14:36 (580): No heartbeat from core client for 30 sec - exiting 09:14:37 (580): No heartbeat from core client for 30 sec - exiting 09:14:38 (580): No heartbeat from core client for 30 sec - exiting 09:59:55 (2268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:23 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:11 (2636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:12 (2636): No heartbeat from core client for 30 sec - exiting 13:37:13 (2636): No heartbeat from core client for 30 sec - exiting 13:37:14 (2636): No heartbeat from core client for 30 sec - exiting 13:37:15 (2636): No heartbeat from core client for 30 sec - exiting 13:37:17 (2636): No heartbeat from core client for 30 sec - exiting 13:37:18 (2636): No heartbeat from core client for 30 sec - exiting 13:37:19 (2636): No heartbeat from core client for 30 sec - exiting 13:37:20 (2636): No heartbeat from core client for 30 sec - exiting 13:37:21 (2636): No heartbeat from core client for 30 sec - exiting 13:37:22 (2636): No heartbeat from core client for 30 sec - exiting 13:42:53 (2360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:55 (2360): No heartbeat from core client for 30 sec - exiting 13:42:56 (2360): No heartbeat from core client for 30 sec - exiting 13:42:57 (2360): No heartbeat from core client for 30 sec - exiting 13:42:59 (2360): No heartbeat from core client for 30 sec - exiting 13:43:00 (2360): No heartbeat from core client for 30 sec - exiting 13:43:01 (2360): No heartbeat from core client for 30 sec - exiting 13:43:03 (2360): No heartbeat from core client for 30 sec - exiting 13:43:04 (2360): No heartbeat from core client for 30 sec - exiting 18:45:16 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:17 (3812): No heartbeat from core client for 30 sec - exiting 18:45:19 (3812): No heartbeat from core client for 30 sec - exiting 18:45:20 (3812): No heartbeat from core client for 30 sec - exiting 18:45:21 (3812): No heartbeat from core client for 30 sec - exiting 18:45:23 (3812): No heartbeat from core client for 30 sec - exiting 18:45:24 (3812): No heartbeat from core client for 30 sec - exiting 19:01:23 (2728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:41 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:43 (1684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:51:59 (1924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:36 (1788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:38 (1788): No heartbeat from core client for 30 sec - exiting 23:43:39 (1788): No heartbeat from core client for 30 sec - exiting 23:43:41 (1788): No heartbeat from core client for 30 sec - exiting 23:43:42 (1788): No heartbeat from core client for 30 sec - exiting 01:29:20 (2044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:06 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:07 (3392): No heartbeat from core client for 30 sec - exiting 01:45:08 (3392): No heartbeat from core client for 30 sec - exiting 01:45:09 (3392): No heartbeat from core client for 30 sec - exiting 01:45:10 (3392): No heartbeat from core client for 30 sec - exiting 01:45:11 (3392): No heartbeat from core client for 30 sec - exiting 01:45:12 (3392): No heartbeat from core client for 30 sec - exiting 01:45:13 (3392): No heartbeat from core client for 30 sec - exiting 01:45:14 (3392): No heartbeat from core client for 30 sec - exiting 01:45:16 (3392): No heartbeat from core client for 30 sec - exiting 01:45:17 (3392): No heartbeat from core client for 30 sec - exiting 03:10:16 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:17 (3676): No heartbeat from core client for 30 sec - exiting 03:10:18 (3676): No heartbeat from core client for 30 sec - exiting 03:10:19 (3676): No heartbeat from core client for 30 sec - exiting 03:10:20 (3676): No heartbeat from core client for 30 sec - exiting 03:10:21 (3676): No heartbeat from core client for 30 sec - exiting 03:10:23 (3676): No heartbeat from core client for 30 sec - exiting 03:10:24 (3676): No heartbeat from core client for 30 sec - exiting 03:10:25 (3676): No heartbeat from core client for 30 sec - exiting 03:10:26 (3676): No heartbeat from core client for 30 sec - exiting 03:10:27 (3676): No heartbeat from core client for 30 sec - exiting 04:16:55 (168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:40 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:41 (3452): No heartbeat from core client for 30 sec - exiting 04:27:42 (3452): No heartbeat from core client for 30 sec - exiting 04:27:43 (3452): No heartbeat from core client for 30 sec - exiting 04:27:44 (3452): No heartbeat from core client for 30 sec - exiting 04:27:45 (3452): No heartbeat from core client for 30 sec - exiting 04:27:46 (3452): No heartbeat from core client for 30 sec - exiting 04:27:47 (3452): No heartbeat from core client for 30 sec - exiting 04:27:49 (3452): No heartbeat from core client for 30 sec - exiting 04:27:50 (3452): No heartbeat from core client for 30 sec - exiting 04:27:51 (3452): No heartbeat from core client for 30 sec - exiting 04:27:53 (3452): No heartbeat from core client for 30 sec - exiting 04:38:30 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:09 (2400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:19 (3524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:10 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:22:14 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:30 (1572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:32 (1572): No heartbeat from core client for 30 sec - exiting 06:32:33 (1572): No heartbeat from core client for 30 sec - exiting 06:32:34 (1572): No heartbeat from core client for 30 sec - exiting 06:32:35 (1572): No heartbeat from core client for 30 sec - exiting 06:32:36 (1572): No heartbeat from core client for 30 sec - exiting 06:32:37 (1572): No heartbeat from core client for 30 sec - exiting 06:32:38 (1572): No heartbeat from core client for 30 sec - exiting 06:32:39 (1572): No heartbeat from core client for 30 sec - exiting 06:32:40 (1572): No heartbeat from core client for 30 sec - exiting 06:32:41 (1572): No heartbeat from core client for 30 sec - exiting 06:40:03 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:40:04 (3168): No heartbeat from core client for 30 sec - exiting 06:40:05 (3168): No heartbeat from core client for 30 sec - exiting 06:40:06 (3168): No heartbeat from core client for 30 sec - exiting 06:40:07 (3168): No heartbeat from core client for 30 sec - exiting 06:40:09 (3168): No heartbeat from core client for 30 sec - exiting 06:40:10 (3168): No heartbeat from core client for 30 sec - exiting 06:40:11 (3168): No heartbeat from core client for 30 sec - exiting 06:40:12 (3168): No heartbeat from core client for 30 sec - exiting 06:40:13 (3168): No heartbeat from core client for 30 sec - exiting 06:40:15 (3168): No heartbeat from core client for 30 sec - exiting 07:11:37 (2748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:13 (3520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:14 (3520): No heartbeat from core client for 30 sec - exiting 07:29:15 (3520): No heartbeat from core client for 30 sec - exiting 07:29:16 (3520): No heartbeat from core client for 30 sec - exiting 07:29:17 (3520): No heartbeat from core client for 30 sec - exiting 07:29:18 (3520): No heartbeat from core client for 30 sec - exiting 07:29:19 (3520): No heartbeat from core client for 30 sec - exiting 07:29:20 (3520): No heartbeat from core client for 30 sec - exiting 07:29:22 (3520): No heartbeat from core client for 30 sec - exiting 07:29:23 (3520): No heartbeat from core client for 30 sec - exiting 07:29:24 (3520): No heartbeat from core client for 30 sec - exiting 08:54:28 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:59 (1764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:33 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3j9yko.pji2c10 Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1864, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3j9yko.daj5850 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3732, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2013 07:22:02	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	1,036,800	4,053,082	3.9092
06 Jul 2013 05:31:33	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	1,010,880	3,970,902	3.9282
06 Jul 2013 05:14:26	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	984,960	3,894,116	3.9536
03 Jul 2013 07:15:01	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	959,040	3,815,000	3.9779
02 Jul 2013 12:03:04	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	933,120	3,736,904	4.0047
02 Jul 2013 11:14:14	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	907,200	3,658,777	4.0330
02 Jul 2013 10:17:31	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	881,280	3,582,219	4.0648
02 Jul 2013 09:52:10	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	855,360	3,508,720	4.1020
28 Jun 2013 02:10:22	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	829,440	3,433,583	4.1396
26 Jun 2013 19:40:58	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	803,520	3,359,309	4.1807
25 Jun 2013 15:04:42	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	777,600	3,285,197	4.2248
24 Jun 2013 18:40:04	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	751,680	3,211,399	4.2723
24 Jun 2013 18:40:04	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	725,760	3,138,641	4.3246
22 Jun 2013 18:21:55	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	699,840	3,065,454	4.3802
22 Jun 2013 18:21:55	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	673,920	2,991,154	4.4384
20 Jun 2013 08:07:42	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	648,000	2,916,158	4.5002
19 Jun 2013 06:10:12	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	622,080	2,841,016	4.5670
19 Jun 2013 06:10:12	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	596,160	2,767,797	4.6427
18 Jun 2013 01:50:39	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	570,240	2,694,076	4.7245
16 Jun 2013 22:15:44	1241496	15650266	hadcm3n_3j9y_1980_40_008319498_1	544,320	2,619,693	4.8128