Task 15447989

Name	hadcm3n_zgpm_1880_40_008248866_0
Workunit	8403990
Created	21 Nov 2012, 18:54:49 UTC
Sent	21 Nov 2012, 18:54:57 UTC
Report deadline	21 Feb 2013, 2:22:08 UTC
Received	7 Dec 2012, 8:10:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1205594
Run time	6 days 16 hours 36 min 46 sec
CPU time	4 days 15 hours 30 min 13 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4452, iMonCtr=1 Model crash detected, will try to restart... 09:51:03 (948): No heartbeat from core client for 30 sec - exiting 09:51:04 (948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:27:57 (3844): No heartbeat from core client for 30 sec - exiting 11:27:58 (3844): No heartbeat from core client for 30 sec - exiting 11:27:59 (3844): No heartbeat from core client for 30 sec - exiting 11:28:00 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied 10:02:23 (704): No heartbeat from core client for 30 sec - exiting 10:02:24 (704): No heartbeat from core client for 30 sec - exiting 10:02:25 (704): No heartbeat from core client for 30 sec - exiting 10:02:26 (704): No heartbeat from core client for 30 sec - exiting 10:02:27 (704): No heartbeat from core client for 30 sec - exiting 10:02:28 (704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:05 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied 07:37:35 (4508): No heartbeat from core client for 30 sec - exiting 07:37:36 (4508): No heartbeat from core client for 30 sec - exiting 07:37:37 (4508): No heartbeat from core client for 30 sec - exiting 07:37:38 (4508): No heartbeat from core client for 30 sec - exiting 07:37:39 (4508): No heartbeat from core client for 30 sec - exiting 07:37:40 (4508): No heartbeat from core client for 30 sec - exiting 07:37:41 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:49 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4708, iMonCtr=1 Model crash detected, will try to restart... 07:53:47 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:54 (1400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1580, iMonCtr=1 Model crash detected, will try to restart... 07:38:51 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:52 (4352): No heartbeat from core client for 30 sec - exiting 07:39:59 (1768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4532, iMonCtr=1 Model crash detected, will try to restart... 15:40:47 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:41 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:23:29 (4600): No heartbeat from core client for 30 sec - exiting 07:23:31 (4600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:21 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:22 (4632): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:35:00 (4580): No heartbeat from core client for 30 sec - exiting 20:35:01 (4580): No heartbeat from core client for 30 sec - exiting 20:35:02 (4580): No heartbeat from core client for 30 sec - exiting 20:35:03 (4580): No heartbeat from core client for 30 sec - exiting 20:35:04 (4580): No heartbeat from core client for 30 sec - exiting 20:35:05 (4580): No heartbeat from core client for 30 sec - exiting 20:35:06 (4580): No heartbeat from core client for 30 sec - exiting 20:35:07 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... C14:35:21 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4652, iMonCtr=1 Model crash detected, will try to restart... 07:13:18 (3820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4864, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... 09:20:59 (3948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... 15:21:49 (3868): No heartbeat from core client for 30 sec - exiting 15:21:50 (3868): No heartbeat from core client for 30 sec - exiting 15:21:51 (3868): No heartbeat from core client for 30 sec - exiting 15:21:52 (3868): No heartbeat from core client for 30 sec - exiting 15:21:53 (3868): No heartbeat from core client for 30 sec - exiting 15:21:54 (3868): No heartbeat from core client for 30 sec - exiting 15:21:55 (3868): No heartbeat from core client for 30 sec - exiting 15:21:56 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2304, iMonCtr=1 Model crash detected, will try to restart... 07:18:33 (2272): No heartbeat from core client for 30 sec - exiting 07:18:35 (2272): No heartbeat from core client for 30 sec - exiting 07:18:36 (2272): No heartbeat from core client for 30 sec - exiting 07:18:37 (2272): No heartbeat from core client for 30 sec - exiting 07:18:38 (2272): No heartbeat from core client for 30 sec - exiting 07:18:39 (2272): No heartbeat from core client for 30 sec - exiting 07:18:40 (2272): No heartbeat from core client for 30 sec - exiting 07:18:41 (2272): No heartbeat from core client for 30 sec - exiting 07:18:42 (2272): No heartbeat from core client for 30 sec - exiting 07:18:43 (2272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:38 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:41 (3292): No heartbeat from core client for 30 sec - exiting 08:06:42 (3292): No heartbeat from core client for 30 sec - exiting 08:06:43 (3292): No heartbeat from core client for 30 sec - exiting 08:06:44 (3292): No heartbeat from core client for 30 sec - exiting 08:06:45 (3292): No heartbeat from core client for 30 sec - exiting 08:06:46 (3292): No heartbeat from core client for 30 sec - exiting 08:06:47 (3292): No heartbeat from core client for 30 sec - exiting 08:06:48 (3292): No heartbeat from core client for 30 sec - exiting 08:06:49 (3292): No heartbeat from core client for 30 sec - exiting 08:06:50 (3292): No heartbeat from core client for 30 sec - exiting 08:06:51 (3292): No heartbeat from core client for 30 sec - exiting 08:06:52 (3292): No heartbeat from core client for 30 sec - exiting 08:06:53 (3292): No heartbeat from core client for 30 sec - exiting 08:06:54 (3292): No heartbeat from core client for 30 sec - exiting 08:06:55 (3292): No heartbeat from core client for 30 sec - exiting 08:06:56 (3292): No heartbeat from core client for 30 sec - exiting 08:06:57 (3292): No heartbeat from core client for 30 sec - exiting 08:06:58 (3292): No heartbeat from core client for 30 sec - exiting 08:06:59 (3292): No heartbeat from core client for 30 sec - exiting 08:07:00 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:50 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Dec 2012 06:06:32	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	259,200	401,407	1.5486
05 Dec 2012 16:06:02	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	233,280	360,886	1.5470
04 Dec 2012 14:23:29	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	207,360	321,405	1.5500
03 Dec 2012 03:53:26	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	181,440	281,127	1.5494
29 Nov 2012 05:51:53	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	155,520	241,403	1.5522
27 Nov 2012 12:14:21	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	129,600	200,180	1.5446
26 Nov 2012 11:00:39	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	103,680	158,548	1.5292
25 Nov 2012 10:35:44	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	77,760	116,573	1.4991
24 Nov 2012 06:12:51	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	51,840	75,100	1.4487
23 Nov 2012 11:21:54	1205594	15447989	hadcm3n_zgpm_1880_40_008248866_0	25,920	35,271	1.3608