Task 16272791

Name	hadcm3n_oezx_1900_40_008474560_3
Workunit	8625399
Created	16 Jan 2014, 8:59:58 UTC
Sent	16 Jan 2014, 9:00:05 UTC
Report deadline	17 Apr 2014, 16:27:16 UTC
Received	11 Feb 2014, 8:48:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1193410
Run time	6 days 7 hours 16 min 23 sec
CPU time	6 days 2 hours 24 min 1 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	0.72 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5976, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 17:42:02 (6936): No heartbeat from core client for 30 sec - exiting 17:42:03 (6936): No heartbeat from core client for 30 sec - exiting 09:07:49 (10484): No heartbeat from core client for 30 sec - exiting 09:07:50 (10484): No heartbeat from core client for 30 sec - exiting 09:07:51 (10484): No heartbeat from core client for 30 sec - exiting 09:07:52 (10484): No heartbeat from core client for 30 sec - exiting 09:07:53 (10484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:38:21 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:59:31 (13028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:51:05 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:37:30 (14128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:31 (14128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 13:32:58 (6288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:53:52 (10200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:22:31 (9996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:48:49 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:48:50 (5364): No heartbeat from core client for 30 sec - exiting 22:50:28 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:42:06 (6336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:46 (9636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:54 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:30:55 (13836): No heartbeat from core client for 30 sec - exiting 09:30:57 (13836): No heartbeat from core client for 30 sec - exiting 09:30:58 (13836): No heartbeat from core client for 30 sec - exiting 09:30:59 (13836): No heartbeat from core client for 30 sec - exiting 09:31:00 (13836): No heartbeat from core client for 30 sec - exiting 09:31:01 (13836): No heartbeat from core client for 30 sec - exiting 09:31:02 (13836): No heartbeat from core client for 30 sec - exiting 09:31:03 (13836): No heartbeat from core client for 30 sec - exiting 09:31:04 (13836): No heartbeat from core client for 30 sec - exiting 09:31:05 (13836): No heartbeat from core client for 30 sec - exiting 09:31:06 (13836): No heartbeat from core client for 30 sec - exiting 09:31:07 (13836): No heartbeat from core client for 30 sec - exiting 09:31:08 (13836): No heartbeat from core client for 30 sec - exiting 09:31:09 (13836): No heartbeat from core client for 30 sec - exiting 09:31:10 (13836): No heartbeat from core client for 30 sec - exiting 09:31:11 (13836): No heartbeat from core client for 30 sec - exiting 09:31:12 (13836): No heartbeat from core client for 30 sec - exiting 09:31:13 (13836): No heartbeat from core client for 30 sec - exiting 09:31:14 (13836): No heartbeat from core client for 30 sec - exiting 09:31:15 (13836): No heartbeat from core client for 30 sec - exiting 09:31:16 (13836): No heartbeat from core client for 30 sec - exiting 09:31:17 (13836): No heartbeat from core client for 30 sec - exiting 09:31:18 (13836): No heartbeat from core client for 30 sec - exiting 09:31:19 (13836): No heartbeat from core client for 30 sec - exiting 09:31:20 (13836): No heartbeat from core client for 30 sec - exiting 09:31:21 (13836): No heartbeat from core client for 30 sec - exiting 09:31:22 (13836): No heartbeat from core client for 30 sec - exiting 09:31:23 (13836): No heartbeat from core client for 30 sec - exiting 09:31:24 (13836): No heartbeat from core client for 30 sec - exiting 09:31:25 (13836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:26 (13836): No heartbeat from core client for 30 sec - exiting 09:31:27 (13836): No heartbeat from core client for 30 sec - exiting 09:31:28 (13836): No heartbeat from core client for 30 sec - exiting 09:31:29 (13836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:06:01 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:51:01 (13216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:02 (13216): No heartbeat from core client for 30 sec - exiting 06:51:03 (13216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:14:16 (8996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:52:59 (9296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:56:28 (8132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:41 (9784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:29:26 (10840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:53:40 (8396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:58:46 (8536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:36:23 (2920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:36:03 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:01:42 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:25:21 (13564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:06:06 (11956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:08 (11956): No heartbeat from core client for 30 sec - exiting 15:06:09 (11956): No heartbeat from core client for 30 sec - exiting 15:06:10 (11956): No heartbeat from core client for 30 sec - exiting 15:06:11 (11956): No heartbeat from core client for 30 sec - exiting 15:06:12 (11956): No heartbeat from core client for 30 sec - exiting 15:06:13 (11956): No heartbeat from core client for 30 sec - exiting 15:06:14 (11956): No heartbeat from core client for 30 sec - exiting 15:06:15 (11956): No heartbeat from core client for 30 sec - exiting 15:06:16 (11956): No heartbeat from core client for 30 sec - exiting 15:06:17 (11956): No heartbeat from core client for 30 sec - exiting 15:06:18 (11956): No heartbeat from core client for 30 sec - exiting 15:06:19 (11956): No heartbeat from core client for 30 sec - exiting 15:06:20 (11956): No heartbeat from core client for 30 sec - exiting 15:06:21 (11956): No heartbeat from core client for 30 sec - exiting 15:06:22 (11956): No heartbeat from core client for 30 sec - exiting 15:06:23 (11956): No heartbeat from core client for 30 sec - exiting 15:06:24 (11956): No heartbeat from core client for 30 sec - exiting 15:06:25 (11956): No heartbeat from core client for 30 sec - exiting 15:06:26 (11956): No heartbeat from core client for 30 sec - exiting 15:06:27 (11956): No heartbeat from core client for 30 sec - exiting 15:06:28 (11956): No heartbeat from core client for 30 sec - exiting 15:06:29 (11956): No heartbeat from core client for 30 sec - exiting 15:06:30 (11956): No heartbeat from core client for 30 sec - exiting 15:06:31 (11956): No heartbeat from core client for 30 sec - exiting 15:06:32 (11956): No heartbeat from core client for 30 sec - exiting 15:06:33 (11956): No heartbeat from core client for 30 sec - exiting 15:06:34 (11956): No heartbeat from core client for 30 sec - exiting 15:06:35 (11956): No heartbeat from core client for 30 sec - exiting 15:06:36 (11956): No heartbeat from core client for 30 sec - exiting 15:06:37 (11956): No heartbeat from core client for 30 sec - exiting 15:06:38 (11956): No heartbeat from core client for 30 sec - exiting 15:06:39 (11956): No heartbeat from core client for 30 sec - exiting 15:06:40 (11956): No heartbeat from core client for 30 sec - exiting 15:06:41 (11956): No heartbeat from core client for 30 sec - exiting 15:06:42 (11956): No heartbeat from core client for 30 sec - exiting 15:06:43 (11956): No heartbeat from core client for 30 sec - exiting 15:06:44 (11956): No heartbeat from core client for 30 sec - exiting 15:06:45 (11956): No heartbeat from core client for 30 sec - exiting 15:06:46 (11956): No heartbeat from core client for 30 sec - exiting 15:06:47 (11956): No heartbeat from core client for 30 sec - exiting 15:06:48 (11956): No heartbeat from core client for 30 sec - exiting 15:06:49 (11956): No heartbeat from core client for 30 sec - exiting 15:06:50 (11956): No heartbeat from core client for 30 sec - exiting 15:06:51 (11956): No heartbeat from core client for 30 sec - exiting 15:06:52 (11956): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:17:34 (8856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:56:23 (9992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:46:32 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:40:38 (12516): No heartbeat from core client for 30 sec - exiting 09:40:39 (12516): No heartbeat from core client for 30 sec - exiting 09:40:40 (12516): No heartbeat from core client for 30 sec - exiting 09:40:41 (12516): No heartbeat from core client for 30 sec - exiting 09:40:42 (12516): No heartbeat from core client for 30 sec - exiting 09:40:43 (12516): No heartbeat from core client for 30 sec - exiting 09:40:44 (12516): No heartbeat from core client for 30 sec - exiting 09:40:45 (12516): No heartbeat from core client for 30 sec - exiting 09:40:46 (12516): No heartbeat from core client for 30 sec - exiting 09:40:47 (12516): No heartbeat from core client for 30 sec - exiting 09:40:48 (12516): No heartbeat from core client for 30 sec - exiting 09:40:49 (12516): No heartbeat from core client for 30 sec - exiting 09:40:50 (12516): No heartbeat from core client for 30 sec - exiting 09:40:51 (12516): No heartbeat from core client for 30 sec - exiting 09:40:52 (12516): No heartbeat from core client for 30 sec - exiting 09:40:53 (12516): No heartbeat from core client for 30 sec - exiting 09:40:54 (12516): No heartbeat from core client for 30 sec - exiting 09:40:55 (12516): No heartbeat from core client for 30 sec - exiting 09:40:56 (12516): No heartbeat from core client for 30 sec - exiting 09:40:57 (12516): No heartbeat from core client for 30 sec - exiting 09:40:58 (12516): No heartbeat from core client for 30 sec - exiting 09:40:59 (12516): No heartbeat from core client for 30 sec - exiting 09:41:00 (12516): No heartbeat from core client for 30 sec - exiting 09:41:01 (12516): No heartbeat from core client for 30 sec - exiting 09:41:02 (12516): No heartbeat from core client for 30 sec - exiting 09:41:03 (12516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:32:51 (12060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:52 (12060): No heartbeat from core client for 30 sec - exiting 18:32:53 (12060): No heartbeat from core client for 30 sec - exiting 18:32:54 (12060): No heartbeat from core client for 30 sec - exiting 18:32:55 (12060): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Feb 2014 16:15:45	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	414,720	492,886	1.1885
05 Feb 2014 11:36:25	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	388,800	456,800	1.1749
03 Feb 2014 16:24:40	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	362,880	425,572	1.1728
02 Feb 2014 21:54:08	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	336,960	393,410	1.1675
01 Feb 2014 20:06:08	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	311,040	359,794	1.1567
31 Jan 2014 11:54:51	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	285,120	329,691	1.1563
31 Jan 2014 09:19:00	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	259,200	302,426	1.1668
31 Jan 2014 09:19:00	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	233,280	276,674	1.1860
28 Jan 2014 10:06:38	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	207,360	248,111	1.1965
27 Jan 2014 09:02:40	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	181,440	214,143	1.1802
24 Jan 2014 10:40:07	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	155,520	182,797	1.1754
23 Jan 2014 10:40:41	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	129,600	153,222	1.1823
22 Jan 2014 11:03:52	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	103,680	123,790	1.1940
21 Jan 2014 15:06:31	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	77,760	93,248	1.1992
19 Jan 2014 14:01:24	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	51,840	62,196	1.1998
17 Jan 2014 10:17:31	1193410	16272791	hadcm3n_oezx_1900_40_008474560_3	25,920	31,107	1.2001