Task 15927769

Name	hadcm3n_z9z5_1960_40_008406313_0
Workunit	8557169
Created	20 Aug 2013, 5:15:19 UTC
Sent	20 Aug 2013, 5:20:35 UTC
Report deadline	19 Nov 2013, 12:47:46 UTC
Received	17 Sep 2013, 1:29:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1129462
Run time	10 days 9 hours 25 min 30 sec
CPU time	10 days 4 hours 22 min
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:21:21 (5572): No heartbeat from core client for 30 sec - exiting 12:21:22 (5572): No heartbeat from core client for 30 sec - exiting 12:21:23 (5572): No heartbeat from core client for 30 sec - exiting 12:21:24 (5572): No heartbeat from core client for 30 sec - exiting 12:21:25 (5572): No heartbeat from core client for 30 sec - exiting 12:21:26 (5572): No heartbeat from core client for 30 sec - exiting 12:21:27 (5572): No heartbeat from core client for 30 sec - exiting 12:21:28 (5572): No heartbeat from core client for 30 sec - exiting 12:21:29 (5572): No heartbeat from core client for 30 sec - exiting 12:21:30 (5572): No heartbeat from core client for 30 sec - exiting 12:21:31 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:32 (5572): No heartbeat from core client for 30 sec - exiting 13:35:37 (2596): No heartbeat from core client for 30 sec - exiting 13:35:39 (2596): No heartbeat from core client for 30 sec - exiting 13:35:40 (2596): No heartbeat from core client for 30 sec - exiting 13:35:41 (2596): No heartbeat from core client for 30 sec - exiting 13:35:42 (2596): No heartbeat from core client for 30 sec - exiting 13:35:43 (2596): No heartbeat from core client for 30 sec - exiting 13:35:44 (2596): No heartbeat from core client for 30 sec - exiting 13:35:45 (2596): No heartbeat from core client for 30 sec - exiting 13:35:46 (2596): No heartbeat from core client for 30 sec - exiting 13:35:47 (2596): No heartbeat from core client for 30 sec - exiting 13:35:48 (2596): No heartbeat from core client for 30 sec - exiting 13:35:49 (2596): No heartbeat from core client for 30 sec - exiting 13:35:50 (2596): No heartbeat from core client for 30 sec - exiting 13:35:51 (2596): No heartbeat from core client for 30 sec - exiting 13:35:52 (2596): No heartbeat from core client for 30 sec - exiting 13:35:53 (2596): No heartbeat from core client for 30 sec - exiting 13:35:54 (2596): No heartbeat from core client for 30 sec - exiting 13:35:55 (2596): No heartbeat from core client for 30 sec - exiting 13:35:56 (2596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1448, iMonCtr=1 Model crash detected, will try to restart... 16:47:29 (5480): No heartbeat from core client for 30 sec - exiting 16:47:31 (5480): No heartbeat from core client for 30 sec - exiting 16:47:32 (5480): No heartbeat from core client for 30 sec - exiting 16:47:33 (5480): No heartbeat from core client for 30 sec - exiting 16:47:34 (5480): No heartbeat from core client for 30 sec - exiting 16:47:35 (5480): No heartbeat from core client for 30 sec - exiting 16:47:36 (5480): No heartbeat from core client for 30 sec - exiting 16:47:37 (5480): No heartbeat from core client for 30 sec - exiting 16:47:38 (5480): No heartbeat from core client for 30 sec - exiting 16:47:39 (5480): No heartbeat from core client for 30 sec - exiting 16:47:40 (5480): No heartbeat from core client for 30 sec - exiting 16:47:41 (5480): No heartbeat from core client for 30 sec - exiting 16:47:42 (5480): No heartbeat from core client for 30 sec - exiting 16:47:43 (5480): No heartbeat from core client for 30 sec - exiting 16:47:44 (5480): No heartbeat from core client for 30 sec - exiting 16:47:45 (5480): No heartbeat from core client for 30 sec - exiting 16:47:46 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:47:47 (5480): No heartbeat from core client for 30 sec - exiting 17:19:34 (1212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:13 (5252): No heartbeat from core client for 30 sec - exiting 09:34:14 (5252): No heartbeat from core client for 30 sec - exiting 09:34:15 (5252): No heartbeat from core client for 30 sec - exiting 09:34:17 (5252): No heartbeat from core client for 30 sec - exiting 09:34:18 (5252): No heartbeat from core client for 30 sec - exiting 09:34:19 (5252): No heartbeat from core client for 30 sec - exiting 09:34:20 (5252): No heartbeat from core client for 30 sec - exiting 09:34:21 (5252): No heartbeat from core client for 30 sec - exiting 09:34:22 (5252): No heartbeat from core client for 30 sec - exiting 09:34:23 (5252): No heartbeat from core client for 30 sec - exiting 09:34:24 (5252): No heartbeat from core client for 30 sec - exiting 09:34:25 (5252): No heartbeat from core client for 30 sec - exiting 09:34:26 (5252): No heartbeat from core client for 30 sec - exiting 09:34:27 (5252): No heartbeat from core client for 30 sec - exiting 09:34:28 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4416, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4416, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4416, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=700, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 13:13:22 (4188): No heartbeat from core client for 30 sec - exiting 13:13:23 (4188): No heartbeat from core client for 30 sec - exiting 13:13:24 (4188): No heartbeat from core client for 30 sec - exiting 13:13:25 (4188): No heartbeat from core client for 30 sec - exiting 13:13:26 (4188): No heartbeat from core client for 30 sec - exiting 13:13:27 (4188): No heartbeat from core client for 30 sec - exiting 13:13:28 (4188): No heartbeat from core client for 30 sec - exiting 13:13:29 (4188): No heartbeat from core client for 30 sec - exiting 13:13:30 (4188): No heartbeat from core client for 30 sec - exiting 13:13:31 (4188): No heartbeat from core client for 30 sec - exiting 13:13:32 (4188): No heartbeat from core client for 30 sec - exiting 13:13:33 (4188): No heartbeat from core client for 30 sec - exiting 13:13:34 (4188): No heartbeat from core client for 30 sec - exiting 13:13:35 (4188): No heartbeat from core client for 30 sec - exiting 13:13:36 (4188): No heartbeat from core client for 30 sec - exiting 13:13:37 (4188): No heartbeat from core client for 30 sec - exiting 13:13:38 (4188): No heartbeat from core client for 30 sec - exiting 13:13:39 (4188): No heartbeat from core client for 30 sec - exiting 13:13:40 (4188): No heartbeat from core client for 30 sec - exiting 13:13:41 (4188): No heartbeat from core client for 30 sec - exiting 13:13:42 (4188): No heartbeat from core client for 30 sec - exiting 13:13:43 (4188): No heartbeat from core client for 30 sec - exiting 13:13:44 (4188): No heartbeat from core client for 30 sec - exiting 13:13:45 (4188): No heartbeat from core client for 30 sec - exiting 13:13:46 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5636, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 09:40:28 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 21 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Sep 2013 09:42:02	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	518,400	867,979	1.6743
31 Aug 2013 19:29:42	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	492,480	825,304	1.6758
31 Aug 2013 07:34:09	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	466,560	783,049	1.6783
29 Aug 2013 15:28:05	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	440,640	739,385	1.6780
29 Aug 2013 03:12:57	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	414,720	696,174	1.6787
28 Aug 2013 13:44:45	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	388,800	653,092	1.6798
28 Aug 2013 01:46:02	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	362,880	610,346	1.6819
27 Aug 2013 07:35:27	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	336,960	568,461	1.6870
26 Aug 2013 09:44:38	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	311,040	524,678	1.6869
25 Aug 2013 20:25:30	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	285,120	479,360	1.6813
25 Aug 2013 08:38:02	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	259,200	437,496	1.6879
24 Aug 2013 20:40:49	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	233,280	395,200	1.6941
24 Aug 2013 07:42:55	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	207,360	348,818	1.6822
23 Aug 2013 19:09:02	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	181,440	304,237	1.6768
23 Aug 2013 12:16:41	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	155,520	261,662	1.6825
23 Aug 2013 12:16:41	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	129,600	216,977	1.6742
22 Aug 2013 06:08:41	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	103,680	172,478	1.6636
21 Aug 2013 17:33:42	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	77,760	128,419	1.6515
21 Aug 2013 05:31:00	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	51,840	85,500	1.6493
20 Aug 2013 17:55:57	1129462	15927769	hadcm3n_z9z5_1960_40_008406313_0	25,920	44,359	1.7114