Task 16836157

Name	hadcm3n_8ai4_1980_40_008723031_1
Workunit	8869009
Created	3 Aug 2014, 9:30:52 UTC
Sent	3 Aug 2014, 9:31:10 UTC
Report deadline	2 Nov 2014, 16:58:21 UTC
Received	11 Sep 2014, 4:56:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291095
Run time	27 days 21 hours 40 min 16 sec
CPU time	25 days 14 hours 12 min 33 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	1.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:13:56 (4836): No heartbeat from core client for 30 sec - exiting 11:13:57 (4836): No heartbeat from core client for 30 sec - exiting 11:13:58 (4836): No heartbeat from core client for 30 sec - exiting 11:13:59 (4836): No heartbeat from core client for 30 sec - exiting 11:14:01 (4836): No heartbeat from core client for 30 sec - exiting 11:14:02 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:12:48 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1544, iMonCtr=1 Model crash detected, will try to restart... 12:20:02 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:46 (3936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:23 (2200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 20:51:45 (4388): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_8ai4_1980_40_008723031/dataout/atmos_restart.day BUFFIN: C I/O Error ferror - Unit 21 - Return code = 32 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Sep 2014 20:53:41	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	622,080	2,191,498	3.5229
10 Sep 2014 00:30:34	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	596,160	2,140,703	3.5908
03 Sep 2014 18:02:32	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	570,240	2,033,443	3.5659
02 Sep 2014 12:00:38	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	544,320	1,941,473	3.5668
01 Sep 2014 04:25:17	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	518,400	1,843,590	3.5563
30 Aug 2014 22:34:16	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	492,480	1,750,365	3.5542
29 Aug 2014 16:26:22	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	466,560	1,653,844	3.5448
29 Aug 2014 09:31:42	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	440,640	1,544,977	3.5062
26 Aug 2014 07:26:16	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	414,720	1,439,945	3.4721
24 Aug 2014 18:25:58	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	388,800	1,336,572	3.4377
23 Aug 2014 09:18:09	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	362,880	1,233,841	3.4001
21 Aug 2014 22:11:17	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	336,960	1,134,003	3.3654
20 Aug 2014 15:30:07	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	311,040	1,038,830	3.3399
19 Aug 2014 04:07:10	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	285,120	934,530	3.2777
17 Aug 2014 15:50:18	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	259,200	832,110	3.2103
16 Aug 2014 02:22:59	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	233,280	745,961	3.1977
15 Aug 2014 04:52:38	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	207,360	688,789	3.3217
15 Aug 2014 04:52:38	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	181,440	661,463	3.6456
15 Aug 2014 04:52:38	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	155,520	609,245	3.9175
15 Aug 2014 04:52:38	1291095	16836157	hadcm3n_8ai4_1980_40_008723031_1	129,600	495,574	3.8239