Task 13352977

Name	hadcm3n_o42o_1940_40_007444746_0
Workunit	7642249
Created	9 Sep 2011, 11:17:57 UTC
Sent	9 Sep 2011, 11:25:41 UTC
Report deadline	9 Dec 2011, 18:52:52 UTC
Received	26 Sep 2011, 16:18:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	860288
Run time	13 days 6 hours 26 min 28 sec
CPU time	13 days 6 hours 26 min 28 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	0.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.2.18</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:07:07 (5925): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:18 (32230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:08:11 (32740): No heartbeat from core client for 30 sec - exiting 00:09:10 (32740): No heartbeat from core client for 30 sec - exiting 00:09:11 (32740): No heartbeat from core client for 30 sec - exiting 00:09:12 (32740): No heartbeat from core client for 30 sec - exiting 00:09:13 (32740): No heartbeat from core client for 30 sec - exiting 00:09:14 (32740): No heartbeat from core client for 30 sec - exiting 00:09:15 (32740): No heartbeat from core client for 30 sec - exiting 00:09:16 (32740): No heartbeat from core client for 30 sec - exiting 00:09:17 (32740): No heartbeat from core client for 30 sec - exiting 00:09:18 (32740): No heartbeat from core client for 30 sec - exiting 00:09:19 (32740): No heartbeat from core client for 30 sec - exiting 00:09:20 (32740): No heartbeat from core client for 30 sec - exiting 00:09:21 (32740): No heartbeat from core client for 30 sec - exiting 00:09:22 (32740): No heartbeat from core client for 30 sec - exiting 00:09:23 (32740): No heartbeat from core client for 30 sec - exiting 00:09:24 (32740): No heartbeat from core client for 30 sec - exiting 00:09:25 (32740): No heartbeat from core client for 30 sec - exiting 00:09:26 (32740): No heartbeat from core client for 30 sec - exiting 00:09:27 (32740): No heartbeat from core client for 30 sec - exiting 00:09:28 (32740): No heartbeat from core client for 30 sec - exiting 00:09:29 (32740): No heartbeat from core client for 30 sec - exiting 00:09:30 (32740): No heartbeat from core client for 30 sec - exiting 00:09:31 (32740): No heartbeat from core client for 30 sec - exiting 00:09:32 (32740): No heartbeat from core client for 30 sec - exiting 00:09:33 (32740): No heartbeat from core client for 30 sec - exiting 00:09:34 (32740): No heartbeat from core client for 30 sec - exiting 00:09:35 (32740): No heartbeat from core client for 30 sec - exiting 00:09:36 (32740): No heartbeat from core client for 30 sec - exiting 00:09:37 (32740): No heartbeat from core client for 30 sec - exiting 00:09:38 (32740): No heartbeat from core client for 30 sec - exiting 00:09:39 (32740): No heartbeat from core client for 30 sec - exiting 00:09:40 (32740): No heartbeat from core client for 30 sec - exiting 00:09:41 (32740): No heartbeat from core client for 30 sec - exiting 00:09:42 (32740): No heartbeat from core client for 30 sec - exiting 00:09:43 (32740): No heartbeat from core client for 30 sec - exiting 00:09:44 (32740): No heartbeat from core client for 30 sec - exiting 00:09:45 (32740): No heartbeat from core client for 30 sec - exiting 00:09:46 (32740): No heartbeat from core client for 30 sec - exiting 00:09:47 (32740): No heartbeat from core client for 30 sec - exiting 00:09:48 (32740): No heartbeat from core client for 30 sec - exiting 00:09:49 (32740): No heartbeat from core client for 30 sec - exiting 00:09:50 (32740): No heartbeat from core client for 30 sec - exiting 00:09:51 (32740): No heartbeat from core client for 30 sec - exiting 00:09:52 (32740): No heartbeat from core client for 30 sec - exiting 00:10:01 (32740): No heartbeat from core client for 30 sec - exiting 00:10:02 (32740): No heartbeat from core client for 30 sec - exiting 00:10:03 (32740): No heartbeat from core client for 30 sec - exiting 00:10:04 (32740): No heartbeat from core client for 30 sec - exiting 00:10:05 (32740): No heartbeat from core client for 30 sec - exiting 00:10:06 (32740): No heartbeat from core client for 30 sec - exiting 00:10:07 (32740): No heartbeat from core client for 30 sec - exiting 00:10:08 (32740): No heartbeat from core client for 30 sec - exiting 00:10:09 (32740): No heartbeat from core client for 30 sec - exiting 00:10:10 (32740): No heartbeat from core client for 30 sec - exiting 00:10:11 (32740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:51 (13171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:06:52 (13171): No heartbeat from core client for 30 sec - exiting 00:06:53 (13171): No heartbeat from core client for 30 sec - exiting 00:06:54 (13171): No heartbeat from core client for 30 sec - exiting 00:06:55 (13171): No heartbeat from core client for 30 sec - exiting 00:06:56 (13171): No heartbeat from core client for 30 sec - exiting 00:06:57 (13171): No heartbeat from core client for 30 sec - exiting 00:06:58 (13171): No heartbeat from core client for 30 sec - exiting 00:06:59 (13171): No heartbeat from core client for 30 sec - exiting 00:07:00 (13171): No heartbeat from core client for 30 sec - exiting 00:07:01 (13171): No heartbeat from core client for 30 sec - exiting 00:07:02 (13171): No heartbeat from core client for 30 sec - exiting 00:07:03 (13171): No heartbeat from core client for 30 sec - exiting 00:07:04 (13171): No heartbeat from core client for 30 sec - exiting 00:07:05 (13171): No heartbeat from core client for 30 sec - exiting 00:07:06 (13171): No heartbeat from core client for 30 sec - exiting 00:07:07 (13171): No heartbeat from core client for 30 sec - exiting 00:07:08 (13171): No heartbeat from core client for 30 sec - exiting 00:07:09 (13171): No heartbeat from core client for 30 sec - exiting 00:07:10 (13171): No heartbeat from core client for 30 sec - exiting 00:07:11 (13171): No heartbeat from core client for 30 sec - exiting 00:07:12 (13171): No heartbeat from core client for 30 sec - exiting 00:07:13 (13171): No heartbeat from core client for 30 sec - exiting 00:07:14 (13171): No heartbeat from core client for 30 sec - exiting 00:07:15 (13171): No heartbeat from core client for 30 sec - exiting 00:07:16 (13171): No heartbeat from core client for 30 sec - exiting 00:07:17 (13171): No heartbeat from core client for 30 sec - exiting 00:07:18 (13171): No heartbeat from core client for 30 sec - exiting 00:07:19 (13171): No heartbeat from core client for 30 sec - exiting 00:07:20 (13171): No heartbeat from core client for 30 sec - exiting 00:07:21 (13171): No heartbeat from core client for 30 sec - exiting 00:07:22 (13171): No heartbeat from core client for 30 sec - exiting 00:07:23 (13171): No heartbeat from core client for 30 sec - exiting 00:07:24 (13171): No heartbeat from core client for 30 sec - exiting 00:07:25 (13171): No heartbeat from core client for 30 sec - exiting 00:07:26 (13171): No heartbeat from core client for 30 sec - exiting 00:07:27 (13171): No heartbeat from core client for 30 sec - exiting 00:07:28 (13171): No heartbeat from core client for 30 sec - exiting 00:07:29 (13171): No heartbeat from core client for 30 sec - exiting 00:07:30 (13171): No heartbeat from core client for 30 sec - exiting 00:07:31 (13171): No heartbeat from core client for 30 sec - exiting 00:07:32 (13171): No heartbeat from core client for 30 sec - exiting 00:07:33 (13171): No heartbeat from core client for 30 sec - exiting 00:08:09 (13303): No heartbeat from core client for 30 sec - exiting 00:10:13 (13303): No heartbeat from core client for 30 sec - exiting 00:10:14 (13303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:09:05 (28085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:53 (10024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:02 (17694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:08:03 (17694): No heartbeat from core client for 30 sec - exiting 00:08:04 (17694): No heartbeat from core client for 30 sec - exiting 00:08:05 (17694): No heartbeat from core client for 30 sec - exiting 00:08:06 (17694): No heartbeat from core client for 30 sec - exiting 00:08:07 (17694): No heartbeat from core client for 30 sec - exiting 00:08:08 (17694): No heartbeat from core client for 30 sec - exiting 00:08:09 (17694): No heartbeat from core client for 30 sec - exiting 00:08:10 (17694): No heartbeat from core client for 30 sec - exiting 00:08:11 (17694): No heartbeat from core client for 30 sec - exiting 00:08:12 (17694): No heartbeat from core client for 30 sec - exiting 00:08:13 (17694): No heartbeat from core client for 30 sec - exiting 00:08:14 (17694): No heartbeat from core client for 30 sec - exiting 00:08:15 (17694): No heartbeat from core client for 30 sec - exiting 00:08:16 (17694): No heartbeat from core client for 30 sec - exiting 00:08:17 (17694): No heartbeat from core client for 30 sec - exiting 00:08:18 (17694): No heartbeat from core client for 30 sec - exiting 00:08:19 (17694): No heartbeat from core client for 30 sec - exiting 00:08:20 (17694): No heartbeat from core client for 30 sec - exiting 00:08:21 (17694): No heartbeat from core client for 30 sec - exiting 00:08:22 (17694): No heartbeat from core client for 30 sec - exiting 00:08:23 (17694): No heartbeat from core client for 30 sec - exiting 00:08:24 (17694): No heartbeat from core client for 30 sec - exiting 00:08:25 (17694): No heartbeat from core client for 30 sec - exiting 00:08:26 (17694): No heartbeat from core client for 30 sec - exiting 00:08:27 (17694): No heartbeat from core client for 30 sec - exiting 00:08:28 (17694): No heartbeat from core client for 30 sec - exiting 00:08:29 (17694): No heartbeat from core client for 30 sec - exiting 00:08:30 (17694): No heartbeat from core client for 30 sec - exiting 00:08:31 (17694): No heartbeat from core client for 30 sec - exiting 00:08:32 (17694): No heartbeat from core client for 30 sec - exiting 00:08:33 (17694): No heartbeat from core client for 30 sec - exiting 00:08:34 (17694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:06:28 (13878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:29 (18984): No heartbeat from core client for 30 sec - exiting 00:09:59 (18984): No heartbeat from core client for 30 sec - exiting 00:10:00 (18984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:51 (31483): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:26 (31483): No heartbeat from core client for 30 sec - exiting 00:10:27 (31483): No heartbeat from core client for 30 sec - exiting 00:10:28 (31483): No heartbeat from core client for 30 sec - exiting 00:10:29 (31483): No heartbeat from core client for 30 sec - exiting 00:10:30 (31483): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:28 (10735): No heartbeat from core client for 30 sec - exiting 00:06:29 (10735): No heartbeat from core client for 30 sec - exiting 00:06:30 (10735): No heartbeat from core client for 30 sec - exiting 00:06:31 (10735): No heartbeat from core client for 30 sec - exiting 00:06:32 (10735): No heartbeat from core client for 30 sec - exiting 00:06:33 (10735): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:07:00 (29602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:06 (17737): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Sep 2011 12:13:39	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	207,360	1,060,245	5.1131
22 Sep 2011 04:30:08	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	181,440	930,022	5.1258
20 Sep 2011 11:35:01	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	155,520	798,623	5.1352
18 Sep 2011 20:28:00	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	129,600	665,468	5.1348
16 Sep 2011 23:05:24	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	103,680	531,834	5.1296
15 Sep 2011 04:49:06	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	77,760	399,001	5.1312
13 Sep 2011 05:00:26	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	51,840	265,884	5.1289
11 Sep 2011 05:01:33	860288	13352977	hadcm3n_o42o_1940_40_007444746_0	25,920	132,544	5.1136