Task 18285536

Name	hadam3p_anz_f6fb_2013_1_009731918_0
Workunit	9803763
Created	8 Apr 2015, 20:41:01 UTC
Sent	11 Apr 2015, 5:45:19 UTC
Report deadline	23 Mar 2016, 11:05:19 UTC
Received	16 Apr 2015, 1:52:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1244719
Run time	4 days 5 hours 49 min 39 sec
CPU time	4 days 3 hours 10 min 19 sec
Validate state	Invalid
Credit	2,497.00
Device peak FLOPS	3.71 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:10:25 (7012): No heartbeat from core client for 30 sec - exiting 08:10:26 (7012): No heartbeat from core client for 30 sec - exiting 08:10:27 (7012): No heartbeat from core client for 30 sec - exiting 08:10:28 (7012): No heartbeat from core client for 30 sec - exiting 08:10:29 (7012): No heartbeat from core client for 30 sec - exiting 08:10:30 (7012): No heartbeat from core client for 30 sec - exiting 08:10:31 (7012): No heartbeat from core client for 30 sec - exiting 08:10:33 (7012): No heartbeat from core client for 30 sec - exiting 08:10:34 (7012): No heartbeat from core client for 30 sec - exiting 08:10:35 (7012): No heartbeat from core client for 30 sec - exiting 08:10:36 (7012): No heartbeat from core client for 30 sec - exiting 08:10:37 (7012): No heartbeat from core client for 30 sec - exiting 08:10:38 (7012): No heartbeat from core client for 30 sec - exiting 08:10:39 (7012): No heartbeat from core client for 30 sec - exiting 08:10:40 (7012): No heartbeat from core client for 30 sec - exiting 08:10:41 (7012): No heartbeat from core client for 30 sec - exiting 08:10:42 (7012): No heartbeat from core client for 30 sec - exiting 08:10:43 (7012): No heartbeat from core client for 30 sec - exiting 08:10:45 (7012): No heartbeat from core client for 30 sec - exiting 08:10:46 (7012): No heartbeat from core client for 30 sec - exiting 08:10:47 (7012): No heartbeat from core client for 30 sec - exiting 08:10:48 (7012): No heartbeat from core client for 30 sec - exiting 08:10:49 (7012): No heartbeat from core client for 30 sec - exiting 08:10:50 (7012): No heartbeat from core client for 30 sec - exiting 08:10:51 (7012): No heartbeat from core client for 30 sec - exiting 08:10:52 (7012): No heartbeat from core client for 30 sec - exiting 08:10:53 (7012): No heartbeat from core client for 30 sec - exiting 08:10:54 (7012): No heartbeat from core client for 30 sec - exiting 08:10:55 (7012): No heartbeat from core client for 30 sec - exiting 08:10:57 (7012): No heartbeat from core client for 30 sec - exiting 08:10:58 (7012): No heartbeat from core client for 30 sec - exiting 08:10:59 (7012): No heartbeat from core client for 30 sec - exiting 08:11:00 (7012): No heartbeat from core client for 30 sec - exiting 08:11:01 (7012): No heartbeat from core client for 30 sec - exiting 08:11:02 (7012): No heartbeat from core client for 30 sec - exiting 08:11:03 (7012): No heartbeat from core client for 30 sec - exiting 08:11:04 (7012): No heartbeat from core client for 30 sec - exiting 08:11:05 (7012): No heartbeat from core client for 30 sec - exiting 08:11:06 (7012): No heartbeat from core client for 30 sec - exiting 08:11:08 (7012): No heartbeat from core client for 30 sec - exiting 08:11:09 (7012): No heartbeat from core client for 30 sec - exiting 08:11:10 (7012): No heartbeat from core client for 30 sec - exiting 08:11:11 (7012): No heartbeat from core client for 30 sec - exiting 08:11:12 (7012): No heartbeat from core client for 30 sec - exiting 08:11:13 (7012): No heartbeat from core client for 30 sec - exiting 08:11:14 (7012): No heartbeat from core client for 30 sec - exiting 08:11:15 (7012): No heartbeat from core client for 30 sec - exiting 08:11:16 (7012): No heartbeat from core client for 30 sec - exiting 08:11:17 (7012): No heartbeat from core client for 30 sec - exiting 08:11:18 (7012): No heartbeat from core client for 30 sec - exiting 08:11:20 (7012): No heartbeat from core client for 30 sec - exiting 08:11:21 (7012): No heartbeat from core client for 30 sec - exiting 08:11:22 (7012): No heartbeat from core client for 30 sec - exiting 08:11:23 (7012): No heartbeat from core client for 30 sec - exiting 08:11:24 (7012): No heartbeat from core client for 30 sec - exiting 08:11:25 (7012): No heartbeat from core client for 30 sec - exiting 08:11:26 (7012): No heartbeat from core client for 30 sec - exiting 08:11:27 (7012): No heartbeat from core client for 30 sec - exiting 08:11:28 (7012): No heartbeat from core client for 30 sec - exiting 08:11:29 (7012): No heartbeat from core client for 30 sec - exiting 08:11:30 (7012): No heartbeat from core client for 30 sec - exiting 08:11:32 (7012): No heartbeat from core client for 30 sec - exiting 08:11:33 (7012): No heartbeat from core client for 30 sec - exiting 08:11:34 (7012): No heartbeat from core client for 30 sec - exiting 08:11:35 (7012): No heartbeat from core client for 30 sec - exiting 08:11:36 (7012): No heartbeat from core client for 30 sec - exiting 08:11:37 (7012): No heartbeat from core client for 30 sec - exiting 08:11:38 (7012): No heartbeat from core client for 30 sec - exiting 08:11:39 (7012): No heartbeat from core client for 30 sec - exiting 08:11:40 (7012): No heartbeat from core client for 30 sec - exiting 08:11:41 (7012): No heartbeat from core client for 30 sec - exiting 08:11:42 (7012): No heartbeat from core client for 30 sec - exiting 08:11:44 (7012): No heartbeat from core client for 30 sec - exiting 08:11:45 (7012): No heartbeat from core client for 30 sec - exiting 08:11:46 (7012): No heartbeat from core client for 30 sec - exiting 08:11:47 (7012): No heartbeat from core client for 30 sec - exiting 08:11:48 (7012): No heartbeat from core client for 30 sec - exiting 08:11:49 (7012): No heartbeat from core client for 30 sec - exiting 08:11:50 (7012): No heartbeat from core client for 30 sec - exiting 08:11:51 (7012): No heartbeat from core client for 30 sec - exiting 08:11:52 (7012): No heartbeat from core client for 30 sec - exiting 08:11:53 (7012): No heartbeat from core client for 30 sec - exiting 08:11:54 (7012): No heartbeat from core client for 30 sec - exiting 08:11:56 (7012): No heartbeat from core client for 30 sec - exiting 08:11:57 (7012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:30:40 (7196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:12 (6652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:56:58 (5428): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6160, iMonCtr=2 Model crash detected, will try to restart... 08:12:24 (2968): No heartbeat from core client for 30 sec - exiting 08:12:25 (2968): No heartbeat from core client for 30 sec - exiting 08:12:26 (2968): No heartbeat from core client for 30 sec - exiting 08:12:27 (2968): No heartbeat from core client for 30 sec - exiting 08:12:28 (2968): No heartbeat from core client for 30 sec - exiting 08:12:29 (2968): No heartbeat from core client for 30 sec - exiting 08:12:30 (2968): No heartbeat from core client for 30 sec - exiting 08:12:31 (2968): No heartbeat from core client for 30 sec - exiting 08:12:32 (2968): No heartbeat from core client for 30 sec - exiting 08:12:34 (2968): No heartbeat from core client for 30 sec - exiting 08:12:35 (2968): No heartbeat from core client for 30 sec - exiting 08:12:36 (2968): No heartbeat from core client for 30 sec - exiting 08:12:37 (2968): No heartbeat from core client for 30 sec - exiting 08:12:38 (2968): No heartbeat from core client for 30 sec - exiting 08:12:39 (2968): No heartbeat from core client for 30 sec - exiting 08:12:40 (2968): No heartbeat from core client for 30 sec - exiting 08:12:41 (2968): No heartbeat from core client for 30 sec - exiting 08:12:42 (2968): No heartbeat from core client for 30 sec - exiting 08:12:43 (2968): No heartbeat from core client for 30 sec - exiting 08:12:44 (2968): No heartbeat from core client for 30 sec - exiting 08:12:46 (2968): No heartbeat from core client for 30 sec - exiting 08:12:47 (2968): No heartbeat from core client for 30 sec - exiting 08:12:48 (2968): No heartbeat from core client for 30 sec - exiting 08:12:49 (2968): No heartbeat from core client for 30 sec - exiting 08:12:50 (2968): No heartbeat from core client for 30 sec - exiting 08:12:51 (2968): No heartbeat from core client for 30 sec - exiting 08:12:52 (2968): No heartbeat from core client for 30 sec - exiting 08:12:53 (2968): No heartbeat from core client for 30 sec - exiting 08:12:54 (2968): No heartbeat from core client for 30 sec - exiting 08:12:55 (2968): No heartbeat from core client for 30 sec - exiting 08:12:56 (2968): No heartbeat from core client for 30 sec - exiting 08:12:58 (2968): No heartbeat from core client for 30 sec - exiting 08:12:59 (2968): No heartbeat from core client for 30 sec - exiting 08:13:00 (2968): No heartbeat from core client for 30 sec - exiting 08:13:01 (2968): No heartbeat from core client for 30 sec - exiting 08:13:02 (2968): No heartbeat from core client for 30 sec - exiting 08:13:03 (2968): No heartbeat from core client for 30 sec - exiting 08:13:04 (2968): No heartbeat from core client for 30 sec - exiting 08:13:05 (2968): No heartbeat from core client for 30 sec - exiting 08:13:06 (2968): No heartbeat from core client for 30 sec - exiting 08:13:07 (2968): No heartbeat from core client for 30 sec - exiting 08:13:08 (2968): No heartbeat from core client for 30 sec - exiting 08:13:10 (2968): No heartbeat from core client for 30 sec - exiting 08:13:11 (2968): No heartbeat from core client for 30 sec - exiting 08:13:12 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4388, selfPID=4388, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:18:37 (1764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6840, iMonCtr=2 Model crash detected, will try to restart... 03:59:44 (8888): No heartbeat from core client for 30 sec - exiting 03:59:46 (8888): No heartbeat from core client for 30 sec - exiting 03:59:47 (8888): No heartbeat from core client for 30 sec - exiting 03:59:48 (8888): No heartbeat from core client for 30 sec - exiting 03:59:49 (8888): No heartbeat from core client for 30 sec - exiting 03:59:50 (8888): No heartbeat from core client for 30 sec - exiting 03:59:51 (8888): No heartbeat from core client for 30 sec - exiting 03:59:52 (8888): No heartbeat from core client for 30 sec - exiting 03:59:53 (8888): No heartbeat from core client for 30 sec - exiting 03:59:54 (8888): No heartbeat from core client for 30 sec - exiting 03:59:56 (8888): No heartbeat from core client for 30 sec - exiting 03:59:57 (8888): No heartbeat from core client for 30 sec - exiting 03:59:58 (8888): No heartbeat from core client for 30 sec - exiting 03:59:59 (8888): No heartbeat from core client for 30 sec - exiting 04:00:00 (8888): No heartbeat from core client for 30 sec - exiting 04:00:01 (8888): No heartbeat from core client for 30 sec - exiting 04:00:02 (8888): No heartbeat from core client for 30 sec - exiting 04:00:03 (8888): No heartbeat from core client for 30 sec - exiting 04:00:04 (8888): No heartbeat from core client for 30 sec - exiting 04:00:05 (8888): No heartbeat from core client for 30 sec - exiting 04:00:06 (8888): No heartbeat from core client for 30 sec - exiting 04:00:08 (8888): No heartbeat from core client for 30 sec - exiting 04:00:09 (8888): No heartbeat from core client for 30 sec - exiting 04:00:10 (8888): No heartbeat from core client for 30 sec - exiting 04:00:11 (8888): No heartbeat from core client for 30 sec - exiting 04:00:12 (8888): No heartbeat from core client for 30 sec - exiting 04:00:13 (8888): No heartbeat from core client for 30 sec - exiting 04:00:14 (8888): No heartbeat from core client for 30 sec - exiting 04:00:15 (8888): No heartbeat from core client for 30 sec - exiting 04:00:16 (8888): No heartbeat from core client for 30 sec - exiting 04:00:17 (8888): No heartbeat from core client for 30 sec - exiting 04:00:18 (8888): No heartbeat from core client for 30 sec - exiting 04:00:20 (8888): No heartbeat from core client for 30 sec - exiting 04:00:21 (8888): No heartbeat from core client for 30 sec - exiting 04:00:22 (8888): No heartbeat from core client for 30 sec - exiting 04:00:23 (8888): No heartbeat from core client for 30 sec - exiting 04:00:24 (8888): No heartbeat from core client for 30 sec - exiting 04:00:25 (8888): No heartbeat from core client for 30 sec - exiting 04:00:26 (8888): No heartbeat from core client for 30 sec - exiting 04:00:27 (8888): No heartbeat from core client for 30 sec - exiting 04:00:28 (8888): No heartbeat from core client for 30 sec - exiting 04:00:29 (8888): No heartbeat from core client for 30 sec - exiting 04:00:30 (8888): No heartbeat from core client for 30 sec - exiting 04:00:32 (8888): No heartbeat from core client for 30 sec - exiting 04:00:33 (8888): No heartbeat from core client for 30 sec - exiting 04:00:34 (8888): No heartbeat from core client for 30 sec - exiting 04:00:35 (8888): No heartbeat from core client for 30 sec - exiting 04:00:36 (8888): No heartbeat from core client for 30 sec - exiting 04:00:37 (8888): No heartbeat from core client for 30 sec - exiting 04:00:38 (8888): No heartbeat from core client for 30 sec - exiting 04:00:39 (8888): No heartbeat from core client for 30 sec - exiting 04:00:40 (8888): No heartbeat from core client for 30 sec - exiting 04:00:41 (8888): No heartbeat from core client for 30 sec - exiting 04:00:42 (8888): No heartbeat from core client for 30 sec - exiting 04:00:44 (8888): No heartbeat from core client for 30 sec - exiting 04:00:45 (8888): No heartbeat from core client for 30 sec - exiting 04:00:46 (8888): No heartbeat from core client for 30 sec - exiting 04:00:47 (8888): No heartbeat from core client for 30 sec - exiting 04:00:48 (8888): No heartbeat from core client for 30 sec - exiting 04:00:49 (8888): No heartbeat from core client for 30 sec - exiting 04:00:50 (8888): No heartbeat from core client for 30 sec - exiting 04:00:51 (8888): No heartbeat from core client for 30 sec - exiting 04:00:52 (8888): No heartbeat from core client for 30 sec - exiting 04:00:53 (8888): No heartbeat from core client for 30 sec - exiting 04:00:54 (8888): No heartbeat from core client for 30 sec - exiting 04:00:56 (8888): No heartbeat from core client for 30 sec - exiting 04:00:57 (8888): No heartbeat from core client for 30 sec - exiting 04:00:58 (8888): No heartbeat from core client for 30 sec - exiting 04:00:59 (8888): No heartbeat from core client for 30 sec - exiting 04:01:00 (8888): No heartbeat from core client for 30 sec - exiting 04:01:01 (8888): No heartbeat from core client for 30 sec - exiting 04:01:02 (8888): No heartbeat from core client for 30 sec - exiting 04:01:03 (8888): No heartbeat from core client for 30 sec - exiting 04:01:04 (8888): No heartbeat from core client for 30 sec - exiting 04:01:05 (8888): No heartbeat from core client for 30 sec - exiting 04:01:06 (8888): No heartbeat from core client for 30 sec - exiting 04:01:08 (8888): No heartbeat from core client for 30 sec - exiting 04:01:09 (8888): No heartbeat from core client for 30 sec - exiting 04:01:10 (8888): No heartbeat from core client for 30 sec - exiting 04:01:11 (8888): No heartbeat from core client for 30 sec - exiting 04:01:12 (8888): No heartbeat from core client for 30 sec - exiting 04:01:13 (8888): No heartbeat from core client for 30 sec - exiting 04:01:14 (8888): No heartbeat from core client for 30 sec - exiting 04:01:15 (8888): No heartbeat from core client for 30 sec - exiting 04:01:16 (8888): No heartbeat from core client for 30 sec - exiting 04:01:17 (8888): No heartbeat from core client for 30 sec - exiting 04:01:18 (8888): No heartbeat from core client for 30 sec - exiting 04:01:20 (8888): No heartbeat from core client for 30 sec - exiting 04:01:21 (8888): No heartbeat from core client for 30 sec - exiting 04:01:22 (8888): No heartbeat from core client for 30 sec - exiting 04:01:23 (8888): No heartbeat from core client for 30 sec - exiting 04:01:24 (8888): No heartbeat from core client for 30 sec - exiting 04:01:25 (8888): No heartbeat from core client for 30 sec - exiting 04:01:26 (8888): No heartbeat from core client for 30 sec - exiting 04:01:27 (8888): No heartbeat from core client for 30 sec - exiting 04:01:28 (8888): No heartbeat from core client for 30 sec - exiting 04:01:29 (8888): No heartbeat from core client for 30 sec - exiting 04:01:31 (8888): No heartbeat from core client for 30 sec - exiting 04:01:32 (8888): No heartbeat from core client for 30 sec - exiting 04:01:33 (8888): No heartbeat from core client for 30 sec - exiting 04:01:34 (8888): No heartbeat from core client for 30 sec - exiting 04:01:35 (8888): No heartbeat from core client for 30 sec - exiting 04:01:36 (8888): No heartbeat from core client for 30 sec - exiting 04:01:37 (8888): No heartbeat from core client for 30 sec - exiting 04:01:38 (8888): No heartbeat from core client for 30 sec - exiting 04:01:39 (8888): No heartbeat from core client for 30 sec - exiting 04:01:40 (8888): No heartbeat from core client for 30 sec - exiting 04:01:41 (8888): No heartbeat from core client for 30 sec - exiting 04:01:43 (8888): No heartbeat from core client for 30 sec - exiting 04:01:44 (8888): No heartbeat from core client for 30 sec - exiting 04:01:45 (8888): No heartbeat from core client for 30 sec - exiting 04:01:46 (8888): No heartbeat from core client for 30 sec - exiting 04:01:47 (8888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_f6fb_2013_1_009731918_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Apr 2015 15:14:55	1244719	18285536	hadam3p_anz_f6fb_2013_1_009731918_0	57,899	324,228	5.5999
14 Apr 2015 18:37:48	1244719	18285536	hadam3p_anz_f6fb_2013_1_009731918_0	46,379	259,239	5.5896
13 Apr 2015 17:04:49	1244719	18285536	hadam3p_anz_f6fb_2013_1_009731918_0	34,859	194,120	5.5687
12 Apr 2015 21:09:23	1244719	18285536	hadam3p_anz_f6fb_2013_1_009731918_0	23,339	128,942	5.5247
12 Apr 2015 01:41:33	1244719	18285536	hadam3p_anz_f6fb_2013_1_009731918_0	11,819	63,595	5.3807