Task 13019002

Name	hadcm3n_t42f_1940_40_007312535_2
Workunit	7509965
Created	28 Jun 2011, 10:02:15 UTC
Sent	28 Jun 2011, 10:02:37 UTC
Report deadline	27 Sep 2011, 17:29:48 UTC
Received	22 Sep 2011, 9:49:34 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1091933
Run time	23 days 12 hours 28 min 38 sec
CPU time	22 days 14 hours 21 min 3 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4060, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2536, iMonCtr=1 Model crash detected, will try to restart... 07:07:30 (664): No heartbeat from core client for 30 sec - exiting 07:07:31 (664): No heartbeat from core client for 30 sec - exiting 07:07:32 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... 07:00:22 (2732): No heartbeat from core client for 30 sec - exiting 07:00:23 (2732): No heartbeat from core client for 30 sec - exiting 07:00:24 (2732): No heartbeat from core client for 30 sec - exiting 07:00:25 (2732): No heartbeat from core client for 30 sec - exiting 07:00:26 (2732): No heartbeat from core client for 30 sec - exiting 07:00:27 (2732): No heartbeat from core client for 30 sec - exiting 07:00:28 (2732): No heartbeat from core client for 30 sec - exiting 07:00:29 (2732): No heartbeat from core client for 30 sec - exiting 07:00:30 (2732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:15 (2468): No heartbeat from core client for 30 sec - exiting 07:00:16 (2468): No heartbeat from core client for 30 sec - exiting 07:00:17 (2468): No heartbeat from core client for 30 sec - exiting 07:00:18 (2468): No heartbeat from core client for 30 sec - exiting 07:00:19 (2468): No heartbeat from core client for 30 sec - exiting 07:00:20 (2468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:31 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3272, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 08:03:06 (3968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1292, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3968, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:05:15 (2148): No heartbeat from core client for 30 sec - exiting 07:05:16 (2148): No heartbeat from core client for 30 sec - exiting 07:05:18 (2148): No heartbeat from core client for 30 sec - exiting 07:05:19 (2148): No heartbeat from core client for 30 sec - exiting 07:05:20 (2148): No heartbeat from core client for 30 sec - exiting 07:05:21 (2148): No heartbeat from core client for 30 sec - exiting 07:05:22 (2148): No heartbeat from core client for 30 sec - exiting 07:05:23 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:24 (2148): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:58:00 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:07:29 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:30 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:45 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:49:26 (3600): No heartbeat from core client for 30 sec - exiting 13:49:27 (3600): No heartbeat from core client for 30 sec - exiting 13:49:29 (3600): No heartbeat from core client for 30 sec - exiting 13:49:30 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:11:16 (2244): No heartbeat from core client for 30 sec - exiting 07:11:18 (2244): No heartbeat from core client for 30 sec - exiting 07:11:19 (2244): No heartbeat from core client for 30 sec - exiting 07:11:20 (2244): No heartbeat from core client for 30 sec - exiting 07:11:21 (2244): No heartbeat from core client for 30 sec - exiting 07:11:22 (2244): No heartbeat from core client for 30 sec - exiting 07:11:23 (2244): No heartbeat from core client for 30 sec - exiting 07:11:24 (2244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t42fko.pjh1c10 Error converting file to netcdf: dataout/t42fko.pih1c10 Error converting file to netcdf: dataout/t42fko.pfh1c10 Error converting file to netcdf: dataout/t42fka.phh1c10 Error converting file to netcdf: dataout/t42fka.pgh1c10 Error converting file to netcdf: dataout/t42fka.peh1c10 Error converting file to netcdf: dataout/t42fka.pdh1c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 07:28:58 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:08:48 (1384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3608, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1740, iMonCtr=1 Model crash detected, will try to restart... 11:24:08 (2296): No heartbeat from core client for 30 sec - exiting 11:24:09 (2296): No heartbeat from core client for 30 sec - exiting 11:24:11 (2296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1036, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=1 Model crash detected, will try to restart... 07:03:59 (1868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... 07:07:35 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:43 (2268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... 09:05:32 (3556): No heartbeat from core client for 30 sec - exiting 09:05:34 (3556): No heartbeat from core client for 30 sec - exiting 09:05:35 (3556): No heartbeat from core client for 30 sec - exiting 09:05:36 (3556): No heartbeat from core client for 30 sec - exiting 09:05:37 (3556): No heartbeat from core client for 30 sec - exiting 09:05:38 (3556): No heartbeat from core client for 30 sec - exiting 09:05:39 (3556): No heartbeat from core client for 30 sec - exiting 09:05:40 (3556): No heartbeat from core client for 30 sec - exiting 09:05:41 (3556): No heartbeat from core client for 30 sec - exiting 09:05:42 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:42:33 (2240): No heartbeat from core client for 30 sec - exiting 07:42:34 (2240): No heartbeat from core client for 30 sec - exiting 07:42:35 (2240): No heartbeat from core client for 30 sec - exiting 07:42:36 (2240): No heartbeat from core client for 30 sec - exiting 07:42:38 (2240): No heartbeat from core client for 30 sec - exiting 07:42:39 (2240): No heartbeat from core client for 30 sec - exiting 07:42:40 (2240): No heartbeat from core client for 30 sec - exiting 07:42:41 (2240): No heartbeat from core client for 30 sec - exiting 07:42:42 (2240): No heartbeat from core client for 30 sec - exiting 07:42:43 (2240): No heartbeat from core client for 30 sec - exiting 07:42:44 (2240): No heartbeat from core client for 30 sec - exiting 07:42:45 (2240): No heartbeat from core client for 30 sec - exiting 07:42:46 (2240): No heartbeat from core client for 30 sec - exiting 07:42:47 (2240): No heartbeat from core client for 30 sec - exiting 07:42:48 (2240): No heartbeat from core client for 30 sec - exiting 07:42:50 (2240): No heartbeat from core client for 30 sec - exiting 07:42:51 (2240): No heartbeat from core client for 30 sec - exiting 07:42:52 (2240): No heartbeat from core client for 30 sec - exiting 07:42:53 (2240): No heartbeat from core client for 30 sec - exiting 07:42:54 (2240): No heartbeat from core client for 30 sec - exiting 07:42:55 (2240): No heartbeat from core client for 30 sec - exiting 07:42:56 (2240): No heartbeat from core client for 30 sec - exiting 07:42:57 (2240): No heartbeat from core client for 30 sec - exiting 07:42:58 (2240): No heartbeat from core client for 30 sec - exiting 07:42:59 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... 15:41:34 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2296, iMonCtr=1 Model crash detected, will try to restart... 07:04:50 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2052, iMonCtr=1 Model crash detected, will try to restart... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Sep 2011 08:47:58	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	1,036,800	1,951,822	1.8825
20 Sep 2011 12:00:19	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	1,010,880	1,904,108	1.8836
15 Sep 2011 10:35:21	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	984,960	1,856,382	1.8847
13 Sep 2011 12:27:55	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	959,040	1,808,754	1.8860
12 Sep 2011 06:11:53	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	933,120	1,760,509	1.8867
07 Sep 2011 13:28:38	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	907,200	1,712,350	1.8875
06 Sep 2011 07:54:37	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	881,280	1,665,061	1.8894
01 Sep 2011 11:11:33	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	855,360	1,617,093	1.8905
30 Aug 2011 11:31:04	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	829,440	1,569,918	1.8927
29 Aug 2011 05:21:02	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	803,520	1,522,832	1.8952
27 Aug 2011 23:23:12	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	777,600	1,476,872	1.8993
25 Aug 2011 09:28:46	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	751,680	1,430,781	1.9034
22 Aug 2011 01:45:03	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	725,760	1,382,144	1.9044
21 Aug 2011 00:59:17	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	699,840	1,333,860	1.9059
20 Aug 2011 11:32:56	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	673,920	1,285,660	1.9077
19 Aug 2011 22:13:52	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	648,000	1,237,570	1.9098
17 Aug 2011 09:34:23	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	622,080	1,188,581	1.9107
15 Aug 2011 12:24:24	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	596,160	1,140,147	1.9125
10 Aug 2011 12:18:04	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	570,240	1,091,701	1.9145
08 Aug 2011 07:09:16	1091933	13019002	hadcm3n_t42f_1940_40_007312535_2	544,320	1,041,396	1.9132