Task 15872995

Name	hadcm3n_zauv_1960_40_008363795_1
Workunit	8514654
Created	29 Jun 2013, 18:10:54 UTC
Sent	29 Jun 2013, 18:26:59 UTC
Report deadline	29 Sep 2013, 1:54:10 UTC
Received	27 Aug 2013, 2:57:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	934531
Run time	35 days 0 hours 0 min 5 sec
CPU time	32 days 8 hours 55 min 31 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16228, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 21:56:12 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3476, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2984, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4044, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4272, iMonCtr=1 Model crash detected, will try to restart... 20:10:06 (2220): No heartbeat from core client for 30 sec - exiting 20:10:07 (2220): No heartbeat from core client for 30 sec - exiting 20:10:08 (2220): No heartbeat from core client for 30 sec - exiting 20:10:09 (2220): No heartbeat from core client for 30 sec - exiting 20:10:10 (2220): No heartbeat from core client for 30 sec - exiting 20:10:11 (2220): No heartbeat from core client for 30 sec - exiting 20:10:12 (2220): No heartbeat from core client for 30 sec - exiting 20:10:13 (2220): No heartbeat from core client for 30 sec - exiting 20:10:15 (2220): No heartbeat from core client for 30 sec - exiting 20:10:16 (2220): No heartbeat from core client for 30 sec - exiting 20:10:17 (2220): No heartbeat from core client for 30 sec - exiting 20:10:18 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:36:09 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:11 (4788): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2700, iMonCtr=1 Model crash detected, will try to restart... 20:15:35 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4840, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3020, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 20:21:01 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:02 (3200): No heartbeat from core client for 30 sec - exiting 20:21:03 (3200): No heartbeat from core client for 30 sec - exiting 20:21:04 (3200): No heartbeat from core client for 30 sec - exiting 20:21:05 (3200): No heartbeat from core client for 30 sec - exiting 20:21:06 (3200): No heartbeat from core client for 30 sec - exiting 20:21:07 (3200): No heartbeat from core client for 30 sec - exiting 20:21:08 (3200): No heartbeat from core client for 30 sec - exiting 20:21:09 (3200): No heartbeat from core client for 30 sec - exiting 20:21:10 (3200): No heartbeat from core client for 30 sec - exiting 21:05:09 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3616, iMonCtr=1 Model crash detected, will try to restart... 22:10:46 (5036): No heartbeat from core client for 30 sec - exiting 22:10:47 (5036): No heartbeat from core client for 30 sec - exiting 22:10:48 (5036): No heartbeat from core client for 30 sec - exiting 22:10:49 (5036): No heartbeat from core client for 30 sec - exiting 22:10:50 (5036): No heartbeat from core client for 30 sec - exiting 22:10:51 (5036): No heartbeat from core client for 30 sec - exiting 22:10:52 (5036): No heartbeat from core client for 30 sec - exiting 22:10:53 (5036): No heartbeat from core client for 30 sec - exiting 22:10:54 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:55 (5036): No heartbeat from core client for 30 sec - exiting 22:10:56 (5036): No heartbeat from core client for 30 sec - exiting 22:10:57 (5036): No heartbeat from core client for 30 sec - exiting 22:10:58 (5036): No heartbeat from core client for 30 sec - exiting 20:13:02 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:03 (3128): No heartbeat from core client for 30 sec - exiting 20:13:04 (3128): No heartbeat from core client for 30 sec - exiting 20:13:05 (3128): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5416, iMonCtr=1 Model crash detected, will try to restart... 20:21:25 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:27 (5936): No heartbeat from core client for 30 sec - exiting 20:36:28 (5936): No heartbeat from core client for 30 sec - exiting 20:36:29 (5936): No heartbeat from core client for 30 sec - exiting 20:36:30 (5936): No heartbeat from core client for 30 sec - exiting 20:36:31 (5936): No heartbeat from core client for 30 sec - exiting 20:36:32 (5936): No heartbeat from core client for 30 sec - exiting 20:36:33 (5936): No heartbeat from core client for 30 sec - exiting 20:36:34 (5936): No heartbeat from core client for 30 sec - exiting 20:36:35 (5936): No heartbeat from core client for 30 sec - exiting 20:36:36 (5936): No heartbeat from core client for 30 sec - exiting 20:36:37 (5936): No heartbeat from core client for 30 sec - exiting 20:36:38 (5936): No heartbeat from core client for 30 sec - exiting 20:36:39 (5936): No heartbeat from core client for 30 sec - exiting 20:36:40 (5936): No heartbeat from core client for 30 sec - exiting 20:36:41 (5936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:42 (4684): No heartbeat from core client for 30 sec - exiting 21:36:43 (4684): No heartbeat from core client for 30 sec - exiting 21:36:44 (4684): No heartbeat from core client for 30 sec - exiting 21:36:45 (4684): No heartbeat from core client for 30 sec - exiting 21:36:46 (4684): No heartbeat from core client for 30 sec - exiting 21:36:47 (4684): No heartbeat from core client for 30 sec - exiting 21:36:48 (4684): No heartbeat from core client for 30 sec - exiting 21:36:49 (4684): No heartbeat from core client for 30 sec - exiting 21:36:50 (4684): No heartbeat from core client for 30 sec - exiting 21:36:51 (4684): No heartbeat from core client for 30 sec - exiting 21:36:52 (4684): No heartbeat from core client for 30 sec - exiting 21:36:53 (4684): No heartbeat from core client for 30 sec - exiting 21:36:54 (4684): No heartbeat from core client for 30 sec - exiting 21:36:55 (4684): No heartbeat from core client for 30 sec - exiting 21:36:56 (4684): No heartbeat from core client for 30 sec - exiting 21:36:57 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:01:47 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... 17:27:10 (4272): No heartbeat from core client for 30 sec - exiting 17:27:11 (4272): No heartbeat from core client for 30 sec - exiting 17:27:12 (4272): No heartbeat from core client for 30 sec - exiting 17:27:13 (4272): No heartbeat from core client for 30 sec - exiting 17:27:14 (4272): No heartbeat from core client for 30 sec - exiting 17:27:15 (4272): No heartbeat from core client for 30 sec - exiting 17:27:16 (4272): No heartbeat from core client for 30 sec - exiting 17:27:17 (4272): No heartbeat from core client for 30 sec - exiting 17:27:18 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:55:23 (3804): No heartbeat from core client for 30 sec - exiting 21:55:24 (3804): No heartbeat from core client for 30 sec - exiting 21:55:25 (3804): No heartbeat from core client for 30 sec - exiting 21:55:26 (3804): No heartbeat from core client for 30 sec - exiting 21:55:27 (3804): No heartbeat from core client for 30 sec - exiting 21:55:28 (3804): No heartbeat from core client for 30 sec - exiting 21:55:29 (3804): No heartbeat from core client for 30 sec - exiting 21:55:30 (3804): No heartbeat from core client for 30 sec - exiting 21:55:31 (3804): No heartbeat from core client for 30 sec - exiting 21:55:32 (3804): No heartbeat from core client for 30 sec - exiting 21:55:33 (3804): No heartbeat from core client for 30 sec - exiting 21:55:34 (3804): No heartbeat from core client for 30 sec - exiting 21:55:35 (3804): No heartbeat from core client for 30 sec - exiting 21:55:36 (3804): No heartbeat from core client for 30 sec - exiting 21:55:37 (3804): No heartbeat from core client for 30 sec - exiting 21:55:38 (3804): No heartbeat from core client for 30 sec - exiting 21:55:39 (3804): No heartbeat from core client for 30 sec - exiting 21:55:40 (3804): No heartbeat from core client for 30 sec - exiting 21:55:41 (3804): No heartbeat from core client for 30 sec - exiting 21:55:42 (3804): No heartbeat from core client for 30 sec - exiting 21:55:43 (3804): No heartbeat from core client for 30 sec - exiting 21:55:44 (3804): No heartbeat from core client for 30 sec - exiting 21:55:45 (3804): No heartbeat from core client for 30 sec - exiting 21:55:46 (3804): No heartbeat from core client for 30 sec - exiting 21:55:47 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:56:22 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:23 (4452): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Aug 2013 02:59:29	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	518,400	2,796,917	5.3953
25 Aug 2013 09:48:16	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	492,480	2,658,213	5.3976
23 Aug 2013 20:59:30	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	466,560	2,520,719	5.4028
18 Aug 2013 08:14:38	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	440,640	2,381,193	5.4039
14 Aug 2013 22:06:18	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	414,720	2,238,921	5.3986
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	388,800	2,096,836	5.3931
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	362,880	1,957,509	5.3944
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	336,960	1,819,198	5.3989
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	311,040	1,680,370	5.4024
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	285,120	1,541,141	5.4052
14 Aug 2013 19:39:24	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	259,200	1,401,139	5.4056
24 Jul 2013 22:45:06	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	233,280	1,255,132	5.3804
23 Jul 2013 20:57:33	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	207,360	1,113,321	5.3690
23 Jul 2013 18:52:59	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	181,440	970,672	5.3498
23 Jul 2013 18:52:59	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	155,520	828,763	5.3290
23 Jul 2013 18:52:59	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	129,600	687,519	5.3049
09 Jul 2013 02:10:19	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	103,680	548,566	5.2910
06 Jul 2013 20:54:55	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	77,760	408,034	5.2474
04 Jul 2013 14:21:47	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	51,840	272,124	5.2493
02 Jul 2013 11:59:54	934531	15872995	hadcm3n_zauv_1960_40_008363795_1	25,920	137,209	5.2936