Task 17317299

Name	hadcm3n_s0b0_1940_40_009093557_1
Workunit	9223893
Created	28 Oct 2014, 21:01:27 UTC
Sent	28 Oct 2014, 21:04:03 UTC
Report deadline	28 Jan 2015, 4:31:14 UTC
Received	20 Nov 2014, 1:46:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1060244
Run time	12 days 2 hours 54 min 57 sec
CPU time	10 days 0 hours 48 min 28 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	2.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> Das Gerät erkennt den Befehl nicht. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:24:06 (8732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:26 (12120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:21 (10728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:37 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:38 (3892): No heartbeat from core client for 30 sec - exiting 18:48:39 (3892): No heartbeat from core client for 30 sec - exiting 18:48:40 (3892): No heartbeat from core client for 30 sec - exiting 18:48:41 (3892): No heartbeat from core client for 30 sec - exiting 18:48:43 (3892): No heartbeat from core client for 30 sec - exiting 18:48:44 (3892): No heartbeat from core client for 30 sec - exiting 18:48:45 (3892): No heartbeat from core client for 30 sec - exiting 18:48:46 (3892): No heartbeat from core client for 30 sec - exiting 18:48:47 (3892): No heartbeat from core client for 30 sec - exiting 18:48:48 (3892): No heartbeat from core client for 30 sec - exiting 03:19:30 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:19:31 (4776): No heartbeat from core client for 30 sec - exiting 03:19:32 (4776): No heartbeat from core client for 30 sec - exiting 03:19:33 (4776): No heartbeat from core client for 30 sec - exiting 03:19:34 (4776): No heartbeat from core client for 30 sec - exiting 03:19:35 (4776): No heartbeat from core client for 30 sec - exiting 03:19:36 (4776): No heartbeat from core client for 30 sec - exiting 03:19:37 (4776): No heartbeat from core client for 30 sec - exiting 03:19:38 (4776): No heartbeat from core client for 30 sec - exiting 03:19:39 (4776): No heartbeat from core client for 30 sec - exiting 03:19:40 (4776): No heartbeat from core client for 30 sec - exiting 06:26:30 (11008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:31 (11008): No heartbeat from core client for 30 sec - exiting 06:26:32 (11008): No heartbeat from core client for 30 sec - exiting 06:26:33 (11008): No heartbeat from core client for 30 sec - exiting 06:26:34 (11008): No heartbeat from core client for 30 sec - exiting 06:26:35 (11008): No heartbeat from core client for 30 sec - exiting 06:26:36 (11008): No heartbeat from core client for 30 sec - exiting 06:26:37 (11008): No heartbeat from core client for 30 sec - exiting 06:26:38 (11008): No heartbeat from core client for 30 sec - exiting 06:26:39 (11008): No heartbeat from core client for 30 sec - exiting 06:26:40 (11008): No heartbeat from core client for 30 sec - exiting 13:24:30 (10852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:31 (10852): No heartbeat from core client for 30 sec - exiting 13:24:32 (10852): No heartbeat from core client for 30 sec - exiting 13:24:33 (10852): No heartbeat from core client for 30 sec - exiting 13:24:34 (10852): No heartbeat from core client for 30 sec - exiting 13:24:35 (10852): No heartbeat from core client for 30 sec - exiting 13:24:36 (10852): No heartbeat from core client for 30 sec - exiting 13:24:37 (10852): No heartbeat from core client for 30 sec - exiting 13:24:38 (10852): No heartbeat from core client for 30 sec - exiting 13:24:39 (10852): No heartbeat from core client for 30 sec - exiting 13:24:40 (10852): No heartbeat from core client for 30 sec - exiting 17:40:03 (11216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:04 (11216): No heartbeat from core client for 30 sec - exiting 17:40:05 (11216): No heartbeat from core client for 30 sec - exiting 17:40:06 (11216): No heartbeat from core client for 30 sec - exiting 17:40:07 (11216): No heartbeat from core client for 30 sec - exiting 17:40:08 (11216): No heartbeat from core client for 30 sec - exiting 17:40:09 (11216): No heartbeat from core client for 30 sec - exiting 17:40:10 (11216): No heartbeat from core client for 30 sec - exiting 17:40:12 (11216): No heartbeat from core client for 30 sec - exiting 17:40:13 (11216): No heartbeat from core client for 30 sec - exiting 17:40:14 (11216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 07:17:53 (8844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:17:54 (8844): No heartbeat from core client for 30 sec - exiting 07:17:55 (8844): No heartbeat from core client for 30 sec - exiting 07:17:56 (8844): No heartbeat from core client for 30 sec - exiting 07:17:57 (8844): No heartbeat from core client for 30 sec - exiting 07:17:58 (8844): No heartbeat from core client for 30 sec - exiting 07:17:59 (8844): No heartbeat from core client for 30 sec - exiting 07:18:00 (8844): No heartbeat from core client for 30 sec - exiting 07:18:01 (8844): No heartbeat from core client for 30 sec - exiting 07:18:02 (8844): No heartbeat from core client for 30 sec - exiting 07:18:03 (8844): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11324, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 23:56:59 (3728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:47:45 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:24:32 (9628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:14 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:01:32 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:13 (3148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:42:55 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:39 (9556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:48 (8656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:49 (8656): No heartbeat from core client for 30 sec - exiting 13:24:50 (8656): No heartbeat from core client for 30 sec - exiting 13:24:51 (8656): No heartbeat from core client for 30 sec - exiting 13:24:52 (8656): No heartbeat from core client for 30 sec - exiting 13:24:53 (8656): No heartbeat from core client for 30 sec - exiting 13:24:54 (8656): No heartbeat from core client for 30 sec - exiting 13:24:55 (8656): No heartbeat from core client for 30 sec - exiting 13:24:56 (8656): No heartbeat from core client for 30 sec - exiting 13:24:57 (8656): No heartbeat from core client for 30 sec - exiting 13:24:58 (8656): No heartbeat from core client for 30 sec - exiting 21:42:32 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:34 (5040): No heartbeat from core client for 30 sec - exiting 21:42:35 (5040): No heartbeat from core client for 30 sec - exiting 21:42:36 (5040): No heartbeat from core client for 30 sec - exiting 21:42:37 (5040): No heartbeat from core client for 30 sec - exiting 21:42:38 (5040): No heartbeat from core client for 30 sec - exiting 21:42:39 (5040): No heartbeat from core client for 30 sec - exiting 21:42:40 (5040): No heartbeat from core client for 30 sec - exiting 21:42:41 (5040): No heartbeat from core client for 30 sec - exiting 21:42:42 (5040): No heartbeat from core client for 30 sec - exiting 01:52:04 (11272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:16 (11372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:17 (11372): No heartbeat from core client for 30 sec - exiting 13:25:18 (11372): No heartbeat from core client for 30 sec - exiting 13:25:19 (11372): No heartbeat from core client for 30 sec - exiting 13:25:20 (11372): No heartbeat from core client for 30 sec - exiting 13:25:21 (11372): No heartbeat from core client for 30 sec - exiting 13:25:22 (11372): No heartbeat from core client for 30 sec - exiting 13:25:23 (11372): No heartbeat from core client for 30 sec - exiting 13:25:24 (11372): No heartbeat from core client for 30 sec - exiting 13:25:25 (11372): No heartbeat from core client for 30 sec - exiting 13:25:26 (11372): No heartbeat from core client for 30 sec - exiting 17:24:49 (7592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:13 (11256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:14 (11256): No heartbeat from core client for 30 sec - exiting 10:20:22 (3760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:23 (3760): No heartbeat from core client for 30 sec - exiting 10:20:24 (3760): No heartbeat from core client for 30 sec - exiting 10:20:26 (3760): No heartbeat from core client for 30 sec - exiting 10:20:27 (3760): No heartbeat from core client for 30 sec - exiting 10:20:28 (3760): No heartbeat from core client for 30 sec - exiting 10:20:29 (3760): No heartbeat from core client for 30 sec - exiting 10:20:30 (3760): No heartbeat from core client for 30 sec - exiting 10:20:31 (3760): No heartbeat from core client for 30 sec - exiting 10:20:32 (3760): No heartbeat from core client for 30 sec - exiting 10:20:33 (3760): No heartbeat from core client for 30 sec - exiting 11:27:48 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:14 (6980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:16 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:17 (4552): No heartbeat from core client for 30 sec - exiting 18:39:19 (4552): No heartbeat from core client for 30 sec - exiting 18:39:20 (4552): No heartbeat from core client for 30 sec - exiting 18:39:21 (4552): No heartbeat from core client for 30 sec - exiting 18:39:22 (4552): No heartbeat from core client for 30 sec - exiting 18:39:23 (4552): No heartbeat from core client for 30 sec - exiting 18:39:24 (4552): No heartbeat from core client for 30 sec - exiting 18:39:25 (4552): No heartbeat from core client for 30 sec - exiting 18:39:26 (4552): No heartbeat from core client for 30 sec - exiting 18:39:27 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:25:10 (10036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:11 (10036): No heartbeat from core client for 30 sec - exiting 23:25:12 (10036): No heartbeat from core client for 30 sec - exiting 23:25:13 (10036): No heartbeat from core client for 30 sec - exiting 23:25:14 (10036): No heartbeat from core client for 30 sec - exiting 23:25:15 (10036): No heartbeat from core client for 30 sec - exiting 23:25:16 (10036): No heartbeat from core client for 30 sec - exiting 23:25:17 (10036): No heartbeat from core client for 30 sec - exiting 23:25:18 (10036): No heartbeat from core client for 30 sec - exiting 23:25:19 (10036): No heartbeat from core client for 30 sec - exiting 23:25:20 (10036): No heartbeat from core client for 30 sec - exiting 23:25:21 (10036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:53:54 (11480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:24:20 (12368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:24:21 (12368): No heartbeat from core client for 30 sec - exiting 21:24:10 (12912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:11 (12912): No heartbeat from core client for 30 sec - exiting 21:24:12 (12912): No heartbeat from core client for 30 sec - exiting 21:24:13 (12912): No heartbeat from core client for 30 sec - exiting 21:24:14 (12912): No heartbeat from core client for 30 sec - exiting 21:24:15 (12912): No heartbeat from core client for 30 sec - exiting 21:24:16 (12912): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:24:11 (1984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:44:56 (10300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:12 (10804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:41 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:24:59 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:00 (4596): No heartbeat from core client for 30 sec - exiting 01:25:01 (4596): No heartbeat from core client for 30 sec - exiting 01:25:02 (4596): No heartbeat from core client for 30 sec - exiting 01:25:03 (4596): No heartbeat from core client for 30 sec - exiting 09:26:24 (12132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:26:25 (12132): No heartbeat from core client for 30 sec - exiting 09:26:26 (12132): No heartbeat from core client for 30 sec - exiting 09:26:27 (12132): No heartbeat from core client for 30 sec - exiting 09:26:28 (12132): No heartbeat from core client for 30 sec - exiting 09:26:29 (12132): No heartbeat from core client for 30 sec - exiting 09:26:30 (12132): No heartbeat from core client for 30 sec - exiting 09:26:31 (12132): No heartbeat from core client for 30 sec - exiting 09:26:32 (12132): No heartbeat from core client for 30 sec - exiting 09:26:33 (12132): No heartbeat from core client for 30 sec - exiting 09:26:34 (12132): No heartbeat from core client for 30 sec - exiting 09:26:36 (12132): No heartbeat from core client for 30 sec - exiting 17:24:40 (6292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:24:41 (6292): No heartbeat from core client for 30 sec - exiting 17:24:42 (6292): No heartbeat from core client for 30 sec - exiting 17:24:43 (6292): No heartbeat from core client for 30 sec - exiting 17:24:44 (6292): No heartbeat from core client for 30 sec - exiting 17:24:45 (6292): No heartbeat from core client for 30 sec - exiting 17:24:46 (6292): No heartbeat from core client for 30 sec - exiting 17:24:47 (6292): No heartbeat from core client for 30 sec - exiting 17:24:48 (6292): No heartbeat from core client for 30 sec - exiting 17:24:49 (6292): No heartbeat from core client for 30 sec - exiting 17:24:50 (6292): No heartbeat from core client for 30 sec - exiting 20:11:41 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:17 (8752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:11:38 (8864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:11:39 (8864): No heartbeat from core client for 30 sec - exiting 20:11:40 (8864): No heartbeat from core client for 30 sec - exiting 20:11:41 (8864): No heartbeat from core client for 30 sec - exiting 20:11:42 (8864): No heartbeat from core client for 30 sec - exiting 20:11:43 (8864): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Nov 2014 09:27:13	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	388,800	831,916	2.1397
18 Nov 2014 20:19:41	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	362,880	783,091	2.1580
18 Nov 2014 04:52:32	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	336,960	733,785	2.1777
17 Nov 2014 08:39:42	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	311,040	681,875	2.1922
16 Nov 2014 18:50:58	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	285,120	633,491	2.2218
13 Nov 2014 14:58:59	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	259,200	581,109	2.2419
12 Nov 2014 09:34:04	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	233,280	531,504	2.2784
10 Nov 2014 09:48:28	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	207,360	474,148	2.2866
05 Nov 2014 01:03:43	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	181,440	418,414	2.3061
03 Nov 2014 23:26:46	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	155,520	362,464	2.3307
03 Nov 2014 01:39:54	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	129,600	301,475	2.3262
02 Nov 2014 04:20:43	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	103,680	239,992	2.3147
31 Oct 2014 18:14:29	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	77,760	175,825	2.2611
30 Oct 2014 21:33:38	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	51,840	115,674	2.2314
29 Oct 2014 22:22:43	1060244	17317299	hadcm3n_s0b0_1940_40_009093557_1	25,920	62,034	2.3933