Task 15934499

Name	hadcm3n_n0vh_1880_40_008410150_0
Workunit	8561006
Created	22 Aug 2013, 4:17:44 UTC
Sent	22 Aug 2013, 15:20:36 UTC
Report deadline	21 Nov 2013, 22:47:47 UTC
Received	14 Sep 2013, 8:00:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1105487
Run time	12 days 3 hours 55 min 11 sec
CPU time	11 days 9 hours 8 min 44 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	3.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 20:45:05 (7104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 21:57:33 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:19:14 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:00 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:01 (3172): No heartbeat from core client for 30 sec - exiting 19:51:59 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:34 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 16:45:20 (5616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:54 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 13:26:08 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:18 (876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:55 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 18:14:00 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:40 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:06:24 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:06:25 (5528): No heartbeat from core client for 30 sec - exiting 14:06:26 (5528): No heartbeat from core client for 30 sec - exiting 14:06:27 (5528): No heartbeat from core client for 30 sec - exiting 14:06:28 (5528): No heartbeat from core client for 30 sec - exiting 14:06:29 (5528): No heartbeat from core client for 30 sec - exiting 17:09:50 (6388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:16 (4496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 21:35:09 (5600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 09:56:15 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:16 (5484): No heartbeat from core client for 30 sec - exiting 09:56:17 (5484): No heartbeat from core client for 30 sec - exiting 09:56:18 (5484): No heartbeat from core client for 30 sec - exiting 09:56:19 (5484): No heartbeat from core client for 30 sec - exiting 11:23:03 (6268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:04 (6268): No heartbeat from core client for 30 sec - exiting 11:23:05 (6268): No heartbeat from core client for 30 sec - exiting 11:23:06 (6268): No heartbeat from core client for 30 sec - exiting 11:23:07 (6268): No heartbeat from core client for 30 sec - exiting 11:23:08 (6268): No heartbeat from core client for 30 sec - exiting 11:23:10 (6268): No heartbeat from core client for 30 sec - exiting 11:23:11 (6268): No heartbeat from core client for 30 sec - exiting 11:23:13 (6268): No heartbeat from core client for 30 sec - exiting 11:23:14 (6268): No heartbeat from core client for 30 sec - exiting 15:25:21 (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 11:15:22 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:24 (4428): No heartbeat from core client for 30 sec - exiting 15:13:10 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 17:29:50 (5640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:30 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n0vhko.pj94c10 Error converting file to netcdf: dataout/n0vhko.pi94c10 Error converting file to netcdf: dataout/n0vhko.pf94c10 Error converting file to netcdf: dataout/n0vhka.ph94c10 Error converting file to netcdf: dataout/n0vhka.pg94c10 Error converting file to netcdf: dataout/n0vhka.pe94c10 Error converting file to netcdf: dataout/n0vhka.pd94c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 16:36:09 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 20:51:01 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:02 (5652): No heartbeat from core client for 30 sec - exiting 20:51:03 (5652): No heartbeat from core client for 30 sec - exiting 20:51:04 (5652): No heartbeat from core client for 30 sec - exiting 20:51:05 (5652): No heartbeat from core client for 30 sec - exiting 20:51:06 (5652): No heartbeat from core client for 30 sec - exiting 20:51:08 (5652): No heartbeat from core client for 30 sec - exiting 20:51:09 (5652): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:26:49 (5136): No heartbeat from core client for 30 sec - exiting 22:26:50 (5136): No heartbeat from core client for 30 sec - exiting 22:26:51 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n0vhko.pj99c10 Error converting file to netcdf: dataout/n0vhko.pi99c10 Error converting file to netcdf: dataout/n0vhko.pf99c10 Error converting file to netcdf: dataout/n0vhka.ph99c10 Error converting file to netcdf: dataout/n0vhka.pg99c10 Error converting file to netcdf: dataout/n0vhka.pe99c10 Error converting file to netcdf: dataout/n0vhka.pd99c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 22:20:19 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5300, iMonCtr=1 Model crash detected, will try to restart... 02:45:51 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 23:47:18 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:47:59 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 18:50:03 (6052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:05 (6052): No heartbeat from core client for 30 sec - exiting 23:21:47 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 07:53:07 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:53:09 (6000): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Sep 2013 22:44:19	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	596,160	979,936	1.6437
12 Sep 2013 22:28:40	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	570,240	935,413	1.6404
10 Sep 2013 20:52:37	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	544,320	891,570	1.6380
09 Sep 2013 20:59:24	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	518,400	846,558	1.6330
08 Sep 2013 22:10:43	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	492,480	801,212	1.6269
08 Sep 2013 09:27:16	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	466,560	757,916	1.6245
07 Sep 2013 10:04:22	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	440,640	713,909	1.6202
06 Sep 2013 13:08:28	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	414,720	669,598	1.6146
05 Sep 2013 11:33:48	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	388,800	624,478	1.6062
04 Sep 2013 02:49:04	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	362,880	579,447	1.5968
03 Sep 2013 09:45:55	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	336,960	534,548	1.5864
02 Sep 2013 12:15:37	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	311,040	490,684	1.5776
01 Sep 2013 12:05:27	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	285,120	448,493	1.5730
31 Aug 2013 16:33:32	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	259,200	407,010	1.5703
30 Aug 2013 20:43:26	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	233,280	370,974	1.5903
29 Aug 2013 20:55:53	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	207,360	353,890	1.7066
28 Aug 2013 19:26:14	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	181,440	310,110	1.7092
27 Aug 2013 18:53:57	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	155,520	266,123	1.7112
26 Aug 2013 18:21:26	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	129,600	222,192	1.7144
25 Aug 2013 20:55:36	1105487	15934499	hadcm3n_n0vh_1880_40_008410150_0	103,680	178,469	1.7213