Task 15543845

Name	hadcm3n_3kzq_1940_40_008265470_1
Workunit	8420594
Created	14 Jan 2013, 16:08:40 UTC
Sent	14 Jan 2013, 16:08:47 UTC
Report deadline	15 Apr 2013, 23:35:58 UTC
Received	14 Mar 2013, 8:44:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1140577
Run time	18 days 13 hours 59 min 55 sec
CPU time	15 days 2 hours 2 min 30 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3kzqko.pje2c10 Error converting file to netcdf: dataout/3kzqko.pie2c10 Error converting file to netcdf: dataout/3kzqko.pfe2c10 Error converting file to netcdf: dataout/3kzqka.phe2c10 Error converting file to netcdf: dataout/3kzqka.pge2c10 Error converting file to netcdf: dataout/3kzqka.pee2c10 Error converting file to netcdf: dataout/3kzqka.pde2c10 Suspended CPDN Monitor - Suspend request from BOINC... 16:08:17 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:51 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:24 (1776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:26 (1776): No heartbeat from core client for 30 sec - exiting 16:19:27 (1776): No heartbeat from core client for 30 sec - exiting 16:19:28 (1776): No heartbeat from core client for 30 sec - exiting 16:19:29 (1776): No heartbeat from core client for 30 sec - exiting 16:19:30 (1776): No heartbeat from core client for 30 sec - exiting 16:19:31 (1776): No heartbeat from core client for 30 sec - exiting 16:19:32 (1776): No heartbeat from core client for 30 sec - exiting 16:19:33 (1776): No heartbeat from core client for 30 sec - exiting 16:19:34 (1776): No heartbeat from core client for 30 sec - exiting 16:19:35 (1776): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5728, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5752, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3144, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3144, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5564, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5756, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5948, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1672, iMonCtr=1 Model crash detected, will try to restart... 17:13:29 (528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4764, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:49:32 (5892): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4112, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5292, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:50:51 (5444): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... 20:58:52 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4284, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4772, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5792, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:55:57 (2072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:44:46 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:44:47 (4732): No heartbeat from core client for 30 sec - exiting 10:44:48 (4732): No heartbeat from core client for 30 sec - exiting 10:44:49 (4732): No heartbeat from core client for 30 sec - exiting 10:44:50 (4732): No heartbeat from core client for 30 sec - exiting 10:44:52 (4732): No heartbeat from core client for 30 sec - exiting 10:44:53 (4732): No heartbeat from core client for 30 sec - exiting 10:44:54 (4732): No heartbeat from core client for 30 sec - exiting 10:44:55 (4732): No heartbeat from core client for 30 sec - exiting 10:44:56 (4732): No heartbeat from core client for 30 sec - exiting 10:44:57 (4732): No heartbeat from core client for 30 sec - exiting 10:46:22 (4564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:37:18 (2440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:19 (2440): No heartbeat from core client for 30 sec - exiting 15:37:20 (2440): No heartbeat from core client for 30 sec - exiting 15:37:21 (2440): No heartbeat from core client for 30 sec - exiting 15:37:22 (2440): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Mar 2013 08:44:49	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	777,600	1,303,336	1.6761
13 Mar 2013 15:35:33	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	751,680	1,262,892	1.6801
13 Mar 2013 10:15:13	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	725,760	1,219,495	1.6803
12 Mar 2013 07:11:06	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	699,840	1,178,443	1.6839
11 Mar 2013 10:43:51	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	673,920	1,136,685	1.6867
08 Mar 2013 12:47:44	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	648,000	1,096,163	1.6916
08 Mar 2013 08:06:17	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	622,080	1,056,094	1.6977
07 Mar 2013 08:35:45	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	596,160	1,015,107	1.7027
04 Mar 2013 08:43:24	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	570,240	973,727	1.7076
28 Feb 2013 15:25:28	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	544,320	925,469	1.7002
27 Feb 2013 11:22:24	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	518,400	878,952	1.6955
27 Feb 2013 09:57:05	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	492,480	833,734	1.6929
26 Feb 2013 06:04:28	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	466,560	786,713	1.6862
22 Feb 2013 18:26:31	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	440,640	739,782	1.6789
22 Feb 2013 10:17:17	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	414,720	692,313	1.6694
20 Feb 2013 10:38:27	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	388,800	644,017	1.6564
18 Feb 2013 09:23:26	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	362,880	594,853	1.6393
13 Feb 2013 17:21:11	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	336,960	546,535	1.6220
11 Feb 2013 15:40:49	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	311,040	500,642	1.6096
07 Feb 2013 17:32:34	1140577	15543845	hadcm3n_3kzq_1940_40_008265470_1	285,120	454,962	1.5957