Task 15821245

Name	hadcm3n_o4sw_1980_40_008385838_0
Workunit	8536697
Created	3 Jun 2013, 4:31:02 UTC
Sent	9 Jun 2013, 8:28:56 UTC
Report deadline	8 Sep 2013, 15:56:07 UTC
Received	3 Jul 2013, 1:06:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1036746
Run time	18 days 21 hours 9 min 59 sec
CPU time	18 days 4 hours 33 min 55 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	1.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 03:36:24 (1788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:41:25 (632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:42:48 (2944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:50:18 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:43 (3064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:09 (2440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:06 (2448): No heartbeat from core client for 30 sec - exiting 06:26:07 (2448): No heartbeat from core client for 30 sec - exiting 06:26:08 (2448): No heartbeat from core client for 30 sec - exiting 06:26:09 (2448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:10 (2448): No heartbeat from core client for 30 sec - exiting 06:26:11 (2448): No heartbeat from core client for 30 sec - exiting 06:26:12 (2448): No heartbeat from core client for 30 sec - exiting 07:26:37 (2064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:44 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:45 (2976): No heartbeat from core client for 30 sec - exiting 17:01:46 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:26:44 (3144): No heartbeat from core client for 30 sec - exiting 14:26:45 (3144): No heartbeat from core client for 30 sec - exiting 14:26:46 (3144): No heartbeat from core client for 30 sec - exiting 14:26:47 (3144): No heartbeat from core client for 30 sec - exiting 14:26:48 (3144): No heartbeat from core client for 30 sec - exiting 14:26:49 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:12 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:35:10 (3136): No heartbeat from core client for 30 sec - exiting 04:35:11 (3136): No heartbeat from core client for 30 sec - exiting 04:35:12 (3136): No heartbeat from core client for 30 sec - exiting 04:35:13 (3136): No heartbeat from core client for 30 sec - exiting 04:35:14 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:02:37 (3372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:39 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:48 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:25 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:16:39 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:49 (3636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:12:53 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:12:54 (2168): No heartbeat from core client for 30 sec - exiting 20:12:55 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:20:49 (4044): No heartbeat from core client for 30 sec - exiting 20:20:50 (4044): No heartbeat from core client for 30 sec - exiting 20:20:51 (4044): No heartbeat from core client for 30 sec - exiting 20:20:52 (4044): No heartbeat from core client for 30 sec - exiting 20:20:53 (4044): No heartbeat from core client for 30 sec - exiting 20:20:55 (4044): No heartbeat from core client for 30 sec - exiting 20:20:56 (4044): No heartbeat from core client for 30 sec - exiting 20:20:57 (4044): No heartbeat from core client for 30 sec - exiting 20:20:58 (4044): No heartbeat from core client for 30 sec - exiting 20:20:59 (4044): No heartbeat from core client for 30 sec - exiting 20:21:00 (4044): No heartbeat from core client for 30 sec - exiting 20:21:01 (4044): No heartbeat from core client for 30 sec - exiting 20:21:02 (4044): No heartbeat from core client for 30 sec - exiting 20:21:03 (4044): No heartbeat from core client for 30 sec - exiting 20:21:04 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:05 (4044): No heartbeat from core client for 30 sec - exiting 20:21:06 (4044): No heartbeat from core client for 30 sec - exiting 20:21:07 (4044): No heartbeat from core client for 30 sec - exiting 21:55:24 (2388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:15:07 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:15:08 (3272): No heartbeat from core client for 30 sec - exiting 05:15:09 (3272): No heartbeat from core client for 30 sec - exiting 05:15:10 (3272): No heartbeat from core client for 30 sec - exiting 05:15:11 (3272): No heartbeat from core client for 30 sec - exiting 05:15:12 (3272): No heartbeat from core client for 30 sec - exiting 09:04:09 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:27:46 (3532): No heartbeat from core client for 30 sec - exiting 16:27:47 (3532): No heartbeat from core client for 30 sec - exiting 16:27:49 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:56 (928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:32:30 (3188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:04 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:13 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:27 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:51 (3580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:27:46 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:27:43 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:34:31 (1204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:16 (428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:32 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:03:50 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:26 (3208): No heartbeat from core client for 30 sec - exiting 18:22:27 (3208): No heartbeat from core client for 30 sec - exiting 18:22:28 (3208): No heartbeat from core client for 30 sec - exiting 18:22:29 (3208): No heartbeat from core client for 30 sec - exiting 18:22:30 (3208): No heartbeat from core client for 30 sec - exiting 18:22:32 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:22:18 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:41:21 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:52:12 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:56 (3460): No heartbeat from core client for 30 sec - exiting 13:03:57 (3460): No heartbeat from core client for 30 sec - exiting 13:03:58 (3460): No heartbeat from core client for 30 sec - exiting 13:03:59 (3460): No heartbeat from core client for 30 sec - exiting 13:04:00 (3460): No heartbeat from core client for 30 sec - exiting 13:04:01 (3460): No heartbeat from core client for 30 sec - exiting 13:04:02 (3460): No heartbeat from core client for 30 sec - exiting 13:04:03 (3460): No heartbeat from core client for 30 sec - exiting 13:04:04 (3460): No heartbeat from core client for 30 sec - exiting 13:04:06 (3460): No heartbeat from core client for 30 sec - exiting 13:04:07 (3460): No heartbeat from core client for 30 sec - exiting 13:04:08 (3460): No heartbeat from core client for 30 sec - exiting 13:04:09 (3460): No heartbeat from core client for 30 sec - exiting 13:04:10 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:26 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:21:44 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:14:23 (3288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:33 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:27:50 (3216): No heartbeat from core client for 30 sec - exiting 17:27:51 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:17:19 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:43:35 (2144): No heartbeat from core client for 30 sec - exiting 15:43:36 (2144): No heartbeat from core client for 30 sec - exiting 15:43:37 (2144): No heartbeat from core client for 30 sec - exiting 15:43:38 (2144): No heartbeat from core client for 30 sec - exiting 15:43:39 (2144): No heartbeat from core client for 30 sec - exiting 15:43:40 (2144): No heartbeat from core client for 30 sec - exiting 15:43:41 (2144): No heartbeat from core client for 30 sec - exiting 15:43:42 (2144): No heartbeat from core client for 30 sec - exiting 15:43:43 (2144): No heartbeat from core client for 30 sec - exiting 15:43:44 (2144): No heartbeat from core client for 30 sec - exiting 15:43:45 (2144): No heartbeat from core client for 30 sec - exiting 15:43:46 (2144): No heartbeat from core client for 30 sec - exiting 15:43:47 (2144): No heartbeat from core client for 30 sec - exiting 15:43:48 (2144): No heartbeat from core client for 30 sec - exiting 15:43:49 (2144): No heartbeat from core client for 30 sec - exiting 15:43:50 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:41 (2856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:09:30 (2920): No heartbeat from core client for 30 sec - exiting 06:09:31 (2920): No heartbeat from core client for 30 sec - exiting 06:09:32 (2920): No heartbeat from core client for 30 sec - exiting 06:09:33 (2920): No heartbeat from core client for 30 sec - exiting 06:09:34 (2920): No heartbeat from core client for 30 sec - exiting 06:09:35 (2920): No heartbeat from core client for 30 sec - exiting 06:09:36 (2920): No heartbeat from core client for 30 sec - exiting 06:09:38 (2920): No heartbeat from core client for 30 sec - exiting 06:09:39 (2920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:10 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:00:11 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:16:58 (2052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:36:43 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:08 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:13 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:16 (3600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:01 (3020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:32:21 (2756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:06:28 (3064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:29 (3064): No heartbeat from core client for 30 sec - exiting 20:14:05 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:31 (2736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:32 (2736): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2013 10:51:03	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	285,120	1,483,377	5.2026
02 Jul 2013 09:47:33	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	259,200	1,347,701	5.1995
26 Jun 2013 18:39:01	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	233,280	1,202,493	5.1547
24 Jun 2013 11:46:13	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	207,360	1,052,364	5.0751
22 Jun 2013 08:01:09	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	181,440	916,341	5.0504
20 Jun 2013 03:10:35	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	155,520	781,665	5.0261
18 Jun 2013 04:36:52	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	129,600	649,341	5.0103
16 Jun 2013 11:37:02	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	103,680	517,803	4.9942
14 Jun 2013 17:18:31	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	77,760	385,328	4.9553
13 Jun 2013 02:46:33	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	51,840	256,247	4.9430
11 Jun 2013 04:16:21	1036746	15821245	hadcm3n_o4sw_1980_40_008385838_0	25,920	128,685	4.9647