Task 14099449

Name	hadcm3n_o7fh_1980_40_007752439_0
Workunit	7907548
Created	15 Feb 2012, 20:23:36 UTC
Sent	15 Feb 2012, 20:23:57 UTC
Report deadline	17 May 2012, 3:51:08 UTC
Received	23 Feb 2012, 4:09:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1113040
Run time	3 days 7 hours 21 min 15 sec
CPU time	2 days 23 hours 25 min 1 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:56:24 (7816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:39 (10156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:33 (7756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:11:00 (10032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:14 (9508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:59 (8724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:09:48 (9428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:16:01 (9228): No heartbeat from core client for 30 sec - exiting 05:16:03 (9228): No heartbeat from core client for 30 sec - exiting 05:16:04 (9228): No heartbeat from core client for 30 sec - exiting 05:16:05 (9228): No heartbeat from core client for 30 sec - exiting 05:16:06 (9228): No heartbeat from core client for 30 sec - exiting 05:16:07 (9228): No heartbeat from core client for 30 sec - exiting 05:16:08 (9228): No heartbeat from core client for 30 sec - exiting 05:16:09 (9228): No heartbeat from core client for 30 sec - exiting 05:16:10 (9228): No heartbeat from core client for 30 sec - exiting 05:16:11 (9228): No heartbeat from core client for 30 sec - exiting 05:16:12 (9228): No heartbeat from core client for 30 sec - exiting 05:16:13 (9228): No heartbeat from core client for 30 sec - exiting 05:16:14 (9228): No heartbeat from core client for 30 sec - exiting 05:16:15 (9228): No heartbeat from core client for 30 sec - exiting 05:16:16 (9228): No heartbeat from core client for 30 sec - exiting 05:16:17 (9228): No heartbeat from core client for 30 sec - exiting 05:16:18 (9228): No heartbeat from core client for 30 sec - exiting 05:16:19 (9228): No heartbeat from core client for 30 sec - exiting 05:16:20 (9228): No heartbeat from core client for 30 sec - exiting 05:16:21 (9228): No heartbeat from core client for 30 sec - exiting 05:16:22 (9228): No heartbeat from core client for 30 sec - exiting 05:16:23 (9228): No heartbeat from core client for 30 sec - exiting 05:16:24 (9228): No heartbeat from core client for 30 sec - exiting 05:16:25 (9228): No heartbeat from core client for 30 sec - exiting 05:16:26 (9228): No heartbeat from core client for 30 sec - exiting 05:16:27 (9228): No heartbeat from core client for 30 sec - exiting 05:16:28 (9228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:25 (9600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:57 (9712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:13 (2548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:54 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:34:54 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:56 (2836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:42:41 (8448): No heartbeat from core client for 30 sec - exiting 05:42:43 (8448): No heartbeat from core client for 30 sec - exiting 05:42:44 (8448): No heartbeat from core client for 30 sec - exiting 05:42:45 (8448): No heartbeat from core client for 30 sec - exiting 05:42:46 (8448): No heartbeat from core client for 30 sec - exiting 05:42:47 (8448): No heartbeat from core client for 30 sec - exiting 05:42:48 (8448): No heartbeat from core client for 30 sec - exiting 05:42:49 (8448): No heartbeat from core client for 30 sec - exiting 05:42:50 (8448): No heartbeat from core client for 30 sec - exiting 05:42:51 (8448): No heartbeat from core client for 30 sec - exiting 05:42:52 (8448): No heartbeat from core client for 30 sec - exiting 05:42:53 (8448): No heartbeat from core client for 30 sec - exiting 05:42:54 (8448): No heartbeat from core client for 30 sec - exiting 05:42:55 (8448): No heartbeat from core client for 30 sec - exiting 05:42:56 (8448): No heartbeat from core client for 30 sec - exiting 05:42:57 (8448): No heartbeat from core client for 30 sec - exiting 05:42:58 (8448): No heartbeat from core client for 30 sec - exiting 05:42:59 (8448): No heartbeat from core client for 30 sec - exiting 05:43:00 (8448): No heartbeat from core client for 30 sec - exiting 05:43:02 (8448): No heartbeat from core client for 30 sec - exiting 05:43:03 (8448): No heartbeat from core client for 30 sec - exiting 05:43:04 (8448): No heartbeat from core client for 30 sec - exiting 05:43:05 (8448): No heartbeat from core client for 30 sec - exiting 05:43:06 (8448): No heartbeat from core client for 30 sec - exiting 05:43:07 (8448): No heartbeat from core client for 30 sec - exiting 05:43:08 (8448): No heartbeat from core client for 30 sec - exiting 05:43:09 (8448): No heartbeat from core client for 30 sec - exiting 05:43:10 (8448): No heartbeat from core client for 30 sec - exiting 05:43:11 (8448): No heartbeat from core client for 30 sec - exiting 05:43:12 (8448): No heartbeat from core client for 30 sec - exiting 05:43:13 (8448): No heartbeat from core client for 30 sec - exiting 05:43:14 (8448): No heartbeat from core client for 30 sec - exiting 05:43:15 (8448): No heartbeat from core client for 30 sec - exiting 05:43:16 (8448): No heartbeat from core client for 30 sec - exiting 05:43:17 (8448): No heartbeat from core client for 30 sec - exiting 05:43:18 (8448): No heartbeat from core client for 30 sec - exiting 05:43:19 (8448): No heartbeat from core client for 30 sec - exiting 05:43:20 (8448): No heartbeat from core client for 30 sec - exiting 05:43:21 (8448): No heartbeat from core client for 30 sec - exiting 05:43:22 (8448): No heartbeat from core client for 30 sec - exiting 05:43:23 (8448): No heartbeat from core client for 30 sec - exiting 05:43:24 (8448): No heartbeat from core client for 30 sec - exiting 05:43:25 (8448): No heartbeat from core client for 30 sec - exiting 05:43:26 (8448): No heartbeat from core client for 30 sec - exiting 05:43:27 (8448): No heartbeat from core client for 30 sec - exiting 05:43:28 (8448): No heartbeat from core client for 30 sec - exiting 05:43:29 (8448): No heartbeat from core client for 30 sec - exiting 05:43:30 (8448): No heartbeat from core client for 30 sec - exiting 05:43:31 (8448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:32 (8448): No heartbeat from core client for 30 sec - exiting 05:45:09 (9876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:45:42 (10168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:38 (5568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:40 (9480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:08:15 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:37 (8484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:07 (5040): No heartbeat from core client for 30 sec - exiting 06:16:08 (5040): No heartbeat from core client for 30 sec - exiting 06:16:09 (5040): No heartbeat from core client for 30 sec - exiting 06:16:10 (5040): No heartbeat from core client for 30 sec - exiting 06:16:11 (5040): No heartbeat from core client for 30 sec - exiting 06:16:12 (5040): No heartbeat from core client for 30 sec - exiting 06:16:13 (5040): No heartbeat from core client for 30 sec - exiting 06:16:14 (5040): No heartbeat from core client for 30 sec - exiting 06:16:15 (5040): No heartbeat from core client for 30 sec - exiting 06:16:16 (5040): No heartbeat from core client for 30 sec - exiting 06:16:17 (5040): No heartbeat from core client for 30 sec - exiting 06:16:18 (5040): No heartbeat from core client for 30 sec - exiting 06:16:19 (5040): No heartbeat from core client for 30 sec - exiting 06:16:20 (5040): No heartbeat from core client for 30 sec - exiting 06:16:21 (5040): No heartbeat from core client for 30 sec - exiting 06:16:22 (5040): No heartbeat from core client for 30 sec - exiting 06:16:23 (5040): No heartbeat from core client for 30 sec - exiting 06:16:24 (5040): No heartbeat from core client for 30 sec - exiting 06:16:25 (5040): No heartbeat from core client for 30 sec - exiting 06:16:26 (5040): No heartbeat from core client for 30 sec - exiting 06:16:27 (5040): No heartbeat from core client for 30 sec - exiting 06:16:28 (5040): No heartbeat from core client for 30 sec - exiting 06:16:29 (5040): No heartbeat from core client for 30 sec - exiting 06:16:30 (5040): No heartbeat from core client for 30 sec - exiting 06:16:31 (5040): No heartbeat from core client for 30 sec - exiting 06:16:32 (5040): No heartbeat from core client for 30 sec - exiting 06:16:33 (5040): No heartbeat from core client for 30 sec - exiting 06:16:35 (5040): No heartbeat from core client for 30 sec - exiting 06:16:36 (5040): No heartbeat from core client for 30 sec - exiting 06:16:37 (5040): No heartbeat from core client for 30 sec - exiting 06:16:38 (5040): No heartbeat from core client for 30 sec - exiting 06:16:39 (5040): No heartbeat from core client for 30 sec - exiting 06:16:40 (5040): No heartbeat from core client for 30 sec - exiting 06:16:41 (5040): No heartbeat from core client for 30 sec - exiting 06:16:42 (5040): No heartbeat from core client for 30 sec - exiting 06:16:43 (5040): No heartbeat from core client for 30 sec - exiting 06:16:44 (5040): No heartbeat from core client for 30 sec - exiting 06:16:45 (5040): No heartbeat from core client for 30 sec - exiting 06:16:46 (5040): No heartbeat from core client for 30 sec - exiting 06:16:47 (5040): No heartbeat from core client for 30 sec - exiting 06:16:48 (5040): No heartbeat from core client for 30 sec - exiting 06:16:49 (5040): No heartbeat from core client for 30 sec - exiting 06:16:50 (5040): No heartbeat from core client for 30 sec - exiting 06:16:51 (5040): No heartbeat from core client for 30 sec - exiting 06:16:52 (5040): No heartbeat from core client for 30 sec - exiting 06:16:53 (5040): No heartbeat from core client for 30 sec - exiting 06:16:54 (5040): No heartbeat from core client for 30 sec - exiting 06:16:55 (5040): No heartbeat from core client for 30 sec - exiting 06:16:56 (5040): No heartbeat from core client for 30 sec - exiting 06:16:57 (5040): No heartbeat from core client for 30 sec - exiting 06:16:58 (5040): No heartbeat from core client for 30 sec - exiting 06:16:59 (5040): No heartbeat from core client for 30 sec - exiting 06:17:00 (5040): No heartbeat from core client for 30 sec - exiting 06:17:01 (5040): No heartbeat from core client for 30 sec - exiting 06:17:02 (5040): No heartbeat from core client for 30 sec - exiting 06:17:03 (5040): No heartbeat from core client for 30 sec - exiting 06:17:04 (5040): No heartbeat from core client for 30 sec - exiting 06:17:05 (5040): No heartbeat from core client for 30 sec - exiting 06:17:06 (5040): No heartbeat from core client for 30 sec - exiting 06:17:07 (5040): No heartbeat from core client for 30 sec - exiting 06:17:08 (5040): No heartbeat from core client for 30 sec - exiting 06:17:09 (5040): No heartbeat from core client for 30 sec - exiting 06:17:10 (5040): No heartbeat from core client for 30 sec - exiting 06:17:11 (5040): No heartbeat from core client for 30 sec - exiting 06:17:12 (5040): No heartbeat from core client for 30 sec - exiting 06:17:13 (5040): No heartbeat from core client for 30 sec - exiting 06:17:14 (5040): No heartbeat from core client for 30 sec - exiting 06:17:15 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:38 (9288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=604, iMonCtr=1 Model crash detected, will try to restart... 19:42:11 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:46 (2400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:43 (2292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:22 (532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:07:58 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:34 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o7fhko.pji8c10 Error converting file to netcdf: dataout/o7fhko.pii8c10 Error converting file to netcdf: dataout/o7fhko.pfi8c10 Error converting file to netcdf: dataout/o7fhka.phi8c10 Error converting file to netcdf: dataout/o7fhka.pgi8c10 Error converting file to netcdf: dataout/o7fhka.pei8c10 Error converting file to netcdf: dataout/o7fhka.pdi8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:03:09 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:50:21 (6296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:07:01 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Feb 2012 04:09:52	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	259,200	257,096	0.9919
22 Feb 2012 19:12:01	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	233,280	232,047	0.9947
22 Feb 2012 09:28:45	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	207,360	206,637	0.9965
22 Feb 2012 01:09:11	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	181,440	180,941	0.9972
21 Feb 2012 16:32:19	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	155,520	155,604	1.0005
20 Feb 2012 23:56:41	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	129,600	130,103	1.0039
20 Feb 2012 16:59:39	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	103,680	104,464	1.0076
20 Feb 2012 06:49:27	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	77,760	78,469	1.0091
19 Feb 2012 19:54:42	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	51,840	52,412	1.0110
19 Feb 2012 10:14:41	1113040	14099449	hadcm3n_o7fh_1980_40_007752439_0	25,920	26,579	1.0254