Task 15926687

Name	hadcm3n_3ida_1980_40_008405328_0
Workunit	8556184
Created	20 Aug 2013, 2:12:38 UTC
Sent	20 Aug 2013, 2:12:50 UTC
Report deadline	19 Nov 2013, 9:40:01 UTC
Received	15 Sep 2013, 20:28:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	3 days 13 hours 24 min 9 sec
CPU time	3 days 4 hours 54 min 26 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 20:05:01 (29884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:18 (29884): No heartbeat from core client for 30 sec - exiting 20:05:19 (29884): No heartbeat from core client for 30 sec - exiting 20:05:20 (29884): No heartbeat from core client for 30 sec - exiting 20:05:21 (29884): No heartbeat from core client for 30 sec - exiting 20:05:23 (29884): No heartbeat from core client for 30 sec - exiting 20:05:24 (29884): No heartbeat from core client for 30 sec - exiting 20:05:25 (29884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:59:30 (36820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:41:38 (55304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:42 (24052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:00:50 (24052): No heartbeat from core client for 30 sec - exiting 23:00:51 (24052): No heartbeat from core client for 30 sec - exiting 23:00:52 (24052): No heartbeat from core client for 30 sec - exiting 23:00:53 (24052): No heartbeat from core client for 30 sec - exiting 23:00:54 (24052): No heartbeat from core client for 30 sec - exiting 23:00:55 (24052): No heartbeat from core client for 30 sec - exiting 23:00:56 (24052): No heartbeat from core client for 30 sec - exiting 23:00:57 (24052): No heartbeat from core client for 30 sec - exiting 23:00:58 (24052): No heartbeat from core client for 30 sec - exiting 23:00:59 (24052): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 23:55:04 (37508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:55:05 (37508): No heartbeat from core client for 30 sec - exiting 23:55:06 (37508): No heartbeat from core client for 30 sec - exiting 23:55:07 (37508): No heartbeat from core client for 30 sec - exiting 23:55:08 (37508): No heartbeat from core client for 30 sec - exiting 23:55:09 (37508): No heartbeat from core client for 30 sec - exiting 23:55:10 (37508): No heartbeat from core client for 30 sec - exiting 23:55:11 (37508): No heartbeat from core client for 30 sec - exiting 23:55:12 (37508): No heartbeat from core client for 30 sec - exiting 23:55:13 (37508): No heartbeat from core client for 30 sec - exiting 00:09:40 (102964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:46 (94076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:37 (63424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:37 (36100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:12 (93084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 03:40:30 (23912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:31 (23912): No heartbeat from core client for 30 sec - exiting 03:40:32 (23912): No heartbeat from core client for 30 sec - exiting 03:40:33 (23912): No heartbeat from core client for 30 sec - exiting 03:40:34 (23912): No heartbeat from core client for 30 sec - exiting 03:40:35 (23912): No heartbeat from core client for 30 sec - exiting 03:40:36 (23912): No heartbeat from core client for 30 sec - exiting 04:40:58 (64700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:58 (85904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:25:04 (85904): No heartbeat from core client for 30 sec - exiting 06:25:05 (85904): No heartbeat from core client for 30 sec - exiting 06:25:06 (85904): No heartbeat from core client for 30 sec - exiting 06:25:07 (85904): No heartbeat from core client for 30 sec - exiting 06:25:08 (85904): No heartbeat from core client for 30 sec - exiting 06:25:09 (85904): No heartbeat from core client for 30 sec - exiting 06:25:10 (85904): No heartbeat from core client for 30 sec - exiting 06:25:11 (85904): No heartbeat from core client for 30 sec - exiting 06:25:12 (85904): No heartbeat from core client for 30 sec - exiting 06:25:13 (85904): No heartbeat from core client for 30 sec - exiting 06:31:16 (85556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:18 (85556): No heartbeat from core client for 30 sec - exiting 08:26:44 (97232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:48 (97232): No heartbeat from core client for 30 sec - exiting 08:26:49 (97232): No heartbeat from core client for 30 sec - exiting 08:26:51 (97232): No heartbeat from core client for 30 sec - exiting 08:26:52 (97232): No heartbeat from core client for 30 sec - exiting 08:26:53 (97232): No heartbeat from core client for 30 sec - exiting 08:26:54 (97232): No heartbeat from core client for 30 sec - exiting 08:26:55 (97232): No heartbeat from core client for 30 sec - exiting 08:26:56 (97232): No heartbeat from core client for 30 sec - exiting 08:26:57 (97232): No heartbeat from core client for 30 sec - exiting 10:38:37 (36368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:46 (36368): No heartbeat from core client for 30 sec - exiting 10:38:47 (36368): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:25:18 (14976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:19 (14976): No heartbeat from core client for 30 sec - exiting 12:25:20 (14976): No heartbeat from core client for 30 sec - exiting 12:25:21 (14976): No heartbeat from core client for 30 sec - exiting 13:17:46 (99584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 14:09:16 (85736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:53 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:07 (39392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:13 (39392): No heartbeat from core client for 30 sec - exiting 15:34:14 (39392): No heartbeat from core client for 30 sec - exiting 15:34:15 (39392): No heartbeat from core client for 30 sec - exiting 15:34:16 (39392): No heartbeat from core client for 30 sec - exiting 15:34:17 (39392): No heartbeat from core client for 30 sec - exiting 15:34:18 (39392): No heartbeat from core client for 30 sec - exiting 15:34:19 (39392): No heartbeat from core client for 30 sec - exiting 15:34:20 (39392): No heartbeat from core client for 30 sec - exiting 15:34:21 (39392): No heartbeat from core client for 30 sec - exiting 15:34:22 (39392): No heartbeat from core client for 30 sec - exiting 17:40:27 (97192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 18:40:12 (66908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:13 (66908): No heartbeat from core client for 30 sec - exiting 18:40:14 (66908): No heartbeat from core client for 30 sec - exiting 18:40:15 (66908): No heartbeat from core client for 30 sec - exiting 18:40:16 (66908): No heartbeat from core client for 30 sec - exiting 18:40:17 (66908): No heartbeat from core client for 30 sec - exiting 18:40:18 (66908): No heartbeat from core client for 30 sec - exiting 19:06:46 (11400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:36:31 (11184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:53:43 (42068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:01 (82176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:33 (49580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:39 (94376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:18 (64720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:26 (64720): No heartbeat from core client for 30 sec - exiting 09:35:07 (49424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:07 (91956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:54 (55524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:55:34 (98900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:42 (92400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:55:46 (41812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:19 (36112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:15:27 (72560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:37 (84716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:55 (84716): No heartbeat from core client for 30 sec - exiting 19:55:56 (84716): No heartbeat from core client for 30 sec - exiting 19:55:57 (84716): No heartbeat from core client for 30 sec - exiting 19:55:58 (84716): No heartbeat from core client for 30 sec - exiting 19:55:59 (84716): No heartbeat from core client for 30 sec - exiting 19:56:00 (84716): No heartbeat from core client for 30 sec - exiting 19:56:01 (84716): No heartbeat from core client for 30 sec - exiting 21:03:14 (34860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:20 (34860): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 21:59:14 (86048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 23:15:23 (56404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:24 (56404): No heartbeat from core client for 30 sec - exiting 23:15:25 (56404): No heartbeat from core client for 30 sec - exiting 23:15:26 (56404): No heartbeat from core client for 30 sec - exiting 23:15:27 (56404): No heartbeat from core client for 30 sec - exiting 23:15:28 (56404): No heartbeat from core client for 30 sec - exiting 00:57:32 (87796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:39 (27992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:29 (30380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:44:15 (54056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:17 (61044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:04 (68708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:17 (68708): No heartbeat from core client for 30 sec - exiting 07:28:18 (68708): No heartbeat from core client for 30 sec - exiting 07:28:19 (68708): No heartbeat from core client for 30 sec - exiting 07:28:20 (68708): No heartbeat from core client for 30 sec - exiting 07:28:21 (68708): No heartbeat from core client for 30 sec - exiting 07:28:22 (68708): No heartbeat from core client for 30 sec - exiting 07:28:23 (68708): No heartbeat from core client for 30 sec - exiting 07:28:24 (68708): No heartbeat from core client for 30 sec - exiting 07:28:25 (68708): No heartbeat from core client for 30 sec - exiting 09:25:00 (17600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:07 (17600): No heartbeat from core client for 30 sec - exiting 09:25:08 (17600): No heartbeat from core client for 30 sec - exiting 09:25:09 (17600): No heartbeat from core client for 30 sec - exiting 09:25:10 (17600): No heartbeat from core client for 30 sec - exiting 09:25:11 (17600): No heartbeat from core client for 30 sec - exiting 09:25:13 (17600): No heartbeat from core client for 30 sec - exiting 09:25:14 (17600): No heartbeat from core client for 30 sec - exiting 09:25:15 (17600): No heartbeat from core client for 30 sec - exiting 10:45:20 (82556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:29 (82556): No heartbeat from core client for 30 sec - exiting 10:45:30 (82556): No heartbeat from core client for 30 sec - exiting 15:18:28 (32404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:09:12 (44844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:19 (17672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 00:53:10 (68180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:53:11 (68180): No heartbeat from core client for 30 sec - exiting 00:53:12 (68180): No heartbeat from core client for 30 sec - exiting 00:53:13 (68180): No heartbeat from core client for 30 sec - exiting 00:53:14 (68180): No heartbeat from core client for 30 sec - exiting 00:53:15 (68180): No heartbeat from core client for 30 sec - exiting 00:53:16 (68180): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:32:06 (71532): No heartbeat from core client for 30 sec - exiting 07:32:07 (71532): No heartbeat from core client for 30 sec - exiting 07:32:08 (71532): No heartbeat from core client for 30 sec - exiting 07:32:09 (71532): No heartbeat from core client for 30 sec - exiting 07:32:10 (71532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... zip error: Could not create output file (was replacing the original zip file) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ida_1980_40_008405328/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Sep 2013 11:34:54	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	259,200	276,896	1.0683
03 Sep 2013 21:06:08	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	233,280	249,272	1.0686
03 Sep 2013 12:42:06	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	207,360	222,047	1.0708
03 Sep 2013 04:08:31	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	181,440	194,919	1.0743
02 Sep 2013 19:44:53	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	155,520	167,990	1.0802
02 Sep 2013 11:25:16	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	129,600	141,019	1.0881
02 Sep 2013 03:22:01	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	103,680	114,350	1.1029
01 Sep 2013 18:53:32	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	77,760	87,121	1.1204
01 Sep 2013 08:44:05	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	51,840	58,543	1.1293
24 Aug 2013 09:38:24	1237173	15926687	hadcm3n_3ida_1980_40_008405328_0	25,920	29,662	1.1444