Task 15803280

Name	hadcm3n_n46i_1880_40_008374657_0
Workunit	8525516
Created	29 May 2013, 22:07:44 UTC
Sent	31 May 2013, 8:22:03 UTC
Report deadline	30 Aug 2013, 15:49:14 UTC
Received	24 Oct 2013, 8:14:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1157323
Run time	23 days 21 hours 17 min 29 sec
CPU time	18 days 1 hours 33 min 49 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.96 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4780, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... 11:14:47 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:33:56 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... 12:53:22 (4440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:54:25 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:34 (4444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:35 (4444): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4344, iMonCtr=1 Model crash detected, will try to restart... 12:54:58 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:03 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... 15:06:51 (2576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4264, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5028, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3304, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4172, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Could not launch model process. Last Error=1450 Called boinc_finish 09:57:31 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:44:18 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:46 (4704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:47:31 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:03 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:41 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:00 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:27:14 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:18 (744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:25:38 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... 10:56:09 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5004, iMonCtr=1 Model crash detected, will try to restart... 16:40:28 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:18 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5032, iMonCtr=1 Model crash detected, will try to restart... 09:53:09 (1768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Oct 2013 08:19:05	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	1,036,800	1,560,824	1.5054
18 Oct 2013 11:31:06	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	1,010,880	1,521,307	1.5049
15 Oct 2013 15:15:10	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	984,960	1,480,817	1.5034
14 Oct 2013 12:24:04	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	959,040	1,441,072	1.5026
11 Oct 2013 10:32:20	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	933,120	1,402,690	1.5032
08 Oct 2013 15:17:12	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	907,200	1,363,783	1.5033
07 Oct 2013 10:46:41	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	881,280	1,323,766	1.5021
02 Oct 2013 15:50:52	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	855,360	1,284,366	1.5016
30 Sep 2013 14:39:45	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	829,440	1,244,728	1.5007
27 Sep 2013 11:53:45	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	803,520	1,205,409	1.5002
23 Sep 2013 10:19:37	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	777,600	1,164,389	1.4974
19 Sep 2013 12:13:10	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	751,680	1,125,867	1.4978
17 Sep 2013 13:37:14	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	725,760	1,083,726	1.4932
16 Sep 2013 07:21:18	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	699,840	1,040,745	1.4871
12 Sep 2013 13:36:08	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	673,920	1,005,369	1.4918
10 Sep 2013 13:00:40	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	648,000	968,668	1.4949
09 Sep 2013 09:08:16	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	622,080	932,281	1.4987
05 Sep 2013 12:13:59	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	596,160	896,071	1.5031
03 Sep 2013 14:58:02	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	570,240	858,255	1.5051
02 Sep 2013 11:30:17	1157323	15803280	hadcm3n_n46i_1880_40_008374657_0	544,320	817,774	1.5024