Task 14359621

Name	hadcm3n_o028_2020_40_007857018_0
Workunit	8012130
Created	4 Apr 2012, 20:28:08 UTC
Sent	4 Apr 2012, 20:30:35 UTC
Report deadline	5 Jul 2012, 3:57:46 UTC
Received	10 May 2012, 3:49:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1089692
Run time	15 days 9 hours 11 min 26 sec
CPU time	11 days 13 hours 44 min 37 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:24:20 (1000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:40:00 (1548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:42:39 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:00 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:35 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:28 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:31 (1576): No heartbeat from core client for 30 sec - exiting 13:19:33 (1576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o028ko.pjm9c10 Error converting file to netcdf: dataout/o028ko.pim9c10 Error converting file to netcdf: dataout/o028ko.pfm9c10 Error converting file to netcdf: dataout/o028ka.phm9c10 Error converting file to netcdf: dataout/o028ka.pgm9c10 Error converting file to netcdf: dataout/o028ka.pem9c10 Error converting file to netcdf: dataout/o028ka.pdm9c10 13:47:07 (2076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:44 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:44 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:33:40 (900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:07 (2096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:26:27 (1884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:00:12 (1996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:07 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:54:44 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:36:51 (936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:58:42 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:15:52 (1396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:04 (2016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:10 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:07 (1896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:30:59 (880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:12:49 (1960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:24 (2036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:17 (1136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:50:19 (2028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:25 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:00 (784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:53 (2188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:47 (2128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:22 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:14 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:08 (1256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:43 (2228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:36 (2036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:29 (544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:46:04 (2052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:33:18 (2128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:26 (1172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:21 (1872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:56 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:11:48 (2124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:42 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:17 (2072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish No Process Handle Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1896, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 May 2012 15:38:08	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	492,480	952,548	1.9342
03 May 2012 08:04:21	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	466,560	904,067	1.9377
02 May 2012 18:31:51	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	440,640	855,607	1.9417
02 May 2012 05:21:30	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	414,720	808,504	1.9495
01 May 2012 17:29:45	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	388,800	764,193	1.9655
30 Apr 2012 13:35:58	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	362,880	717,783	1.9780
29 Apr 2012 05:42:53	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	336,960	672,542	1.9959
28 Apr 2012 08:05:18	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	311,040	623,402	2.0043
27 Apr 2012 08:06:11	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	285,120	572,806	2.0090
26 Apr 2012 08:46:14	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	259,200	526,065	2.0296
25 Apr 2012 17:21:16	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	233,280	477,420	2.0466
24 Apr 2012 16:49:25	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	207,360	423,592	2.0428
23 Apr 2012 15:11:56	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	181,440	369,403	2.0360
22 Apr 2012 12:30:07	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	155,520	315,328	2.0276
21 Apr 2012 10:33:17	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	129,600	263,727	2.0349
20 Apr 2012 11:22:08	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	103,680	208,827	2.0141
19 Apr 2012 03:47:45	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	77,760	157,008	2.0191
18 Apr 2012 04:01:58	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	51,840	100,947	1.9473
17 Apr 2012 06:51:47	1089692	14359621	hadcm3n_o028_2020_40_007857018_0	25,920	48,296	1.8633