Task 12993222

Name	hadcm3n_s6q8_1940_40_007301089_1
Workunit	7498513
Created	21 Jun 2011, 2:12:23 UTC
Sent	21 Jun 2011, 2:13:24 UTC
Report deadline	20 Sep 2011, 9:40:35 UTC
Received	5 Jul 2011, 15:02:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103150
Run time	5 days 11 hours 45 min 3 sec
CPU time	5 days 1 hours 3 min 45 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:12:32 (10356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:33:24 (6184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:22:11 (5636): No heartbeat from core client for 30 sec - exiting 22:22:12 (5636): No heartbeat from core client for 30 sec - exiting 22:22:13 (5636): No heartbeat from core client for 30 sec - exiting 22:22:14 (5636): No heartbeat from core client for 30 sec - exiting 22:22:15 (5636): No heartbeat from core client for 30 sec - exiting 22:22:16 (5636): No heartbeat from core client for 30 sec - exiting 22:22:17 (5636): No heartbeat from core client for 30 sec - exiting 22:22:18 (5636): No heartbeat from core client for 30 sec - exiting 22:22:19 (5636): No heartbeat from core client for 30 sec - exiting 22:22:20 (5636): No heartbeat from core client for 30 sec - exiting 22:22:21 (5636): No heartbeat from core client for 30 sec - exiting 22:22:22 (5636): No heartbeat from core client for 30 sec - exiting 22:22:23 (5636): No heartbeat from core client for 30 sec - exiting 22:22:24 (5636): No heartbeat from core client for 30 sec - exiting 22:22:25 (5636): No heartbeat from core client for 30 sec - exiting 22:22:26 (5636): No heartbeat from core client for 30 sec - exiting 22:22:27 (5636): No heartbeat from core client for 30 sec - exiting 22:22:28 (5636): No heartbeat from core client for 30 sec - exiting 22:22:29 (5636): No heartbeat from core client for 30 sec - exiting 22:22:30 (5636): No heartbeat from core client for 30 sec - exiting 22:22:31 (5636): No heartbeat from core client for 30 sec - exiting 22:22:32 (5636): No heartbeat from core client for 30 sec - exiting 22:22:33 (5636): No heartbeat from core client for 30 sec - exiting 22:22:34 (5636): No heartbeat from core client for 30 sec - exiting 22:22:35 (5636): No heartbeat from core client for 30 sec - exiting 22:22:36 (5636): No heartbeat from core client for 30 sec - exiting 22:22:37 (5636): No heartbeat from core client for 30 sec - exiting 22:22:38 (5636): No heartbeat from core client for 30 sec - exiting 22:22:39 (5636): No heartbeat from core client for 30 sec - exiting 22:22:40 (5636): No heartbeat from core client for 30 sec - exiting 22:22:41 (5636): No heartbeat from core client for 30 sec - exiting 22:22:42 (5636): No heartbeat from core client for 30 sec - exiting 22:22:43 (5636): No heartbeat from core client for 30 sec - exiting 22:22:44 (5636): No heartbeat from core client for 30 sec - exiting 22:22:45 (5636): No heartbeat from core client for 30 sec - exiting 22:22:46 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6572, iMonCtr=1 Model crash detected, will try to restart... 08:35:25 (4184): No heartbeat from core client for 30 sec - exiting 08:35:26 (4184): No heartbeat from core client for 30 sec - exiting 08:35:27 (4184): No heartbeat from core client for 30 sec - exiting 08:35:28 (4184): No heartbeat from core client for 30 sec - exiting 08:35:29 (4184): No heartbeat from core client for 30 sec - exiting 08:35:30 (4184): No heartbeat from core client for 30 sec - exiting 08:35:31 (4184): No heartbeat from core client for 30 sec - exiting 08:35:32 (4184): No heartbeat from core client for 30 sec - exiting 08:35:33 (4184): No heartbeat from core client for 30 sec - exiting 08:35:34 (4184): No heartbeat from core client for 30 sec - exiting 08:35:35 (4184): No heartbeat from core client for 30 sec - exiting 08:35:36 (4184): No heartbeat from core client for 30 sec - exiting 08:35:37 (4184): No heartbeat from core client for 30 sec - exiting 08:35:38 (4184): No heartbeat from core client for 30 sec - exiting 08:35:39 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:40 (4184): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:23:14 (6580): No heartbeat from core client for 30 sec - exiting 09:23:15 (6580): No heartbeat from core client for 30 sec - exiting 09:23:16 (6580): No heartbeat from core client for 30 sec - exiting 09:23:17 (6580): No heartbeat from core client for 30 sec - exiting 09:23:18 (6580): No heartbeat from core client for 30 sec - exiting 09:23:19 (6580): No heartbeat from core client for 30 sec - exiting 09:23:20 (6580): No heartbeat from core client for 30 sec - exiting 09:23:21 (6580): No heartbeat from core client for 30 sec - exiting 09:23:22 (6580): No heartbeat from core client for 30 sec - exiting 09:23:23 (6580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:24 (6580): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:26:18 (5388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:26:19 (5388): No heartbeat from core client for 30 sec - exiting 13:26:20 (5388): No heartbeat from core client for 30 sec - exiting 13:26:21 (5388): No heartbeat from core client for 30 sec - exiting 13:26:22 (5388): No heartbeat from core client for 30 sec - exiting 13:26:23 (5388): No heartbeat from core client for 30 sec - exiting 13:26:24 (5388): No heartbeat from core client for 30 sec - exiting 13:26:25 (5388): No heartbeat from core client for 30 sec - exiting 13:26:26 (5388): No heartbeat from core client for 30 sec - exiting 13:26:27 (5388): No heartbeat from core client for 30 sec - exiting 13:26:28 (5388): No heartbeat from core client for 30 sec - exiting 19:51:29 (7944): No heartbeat from core client for 30 sec - exiting 19:51:30 (7944): No heartbeat from core client for 30 sec - exiting 19:51:31 (7944): No heartbeat from core client for 30 sec - exiting 19:51:32 (7944): No heartbeat from core client for 30 sec - exiting 19:51:33 (7944): No heartbeat from core client for 30 sec - exiting 19:51:34 (7944): No heartbeat from core client for 30 sec - exiting 19:51:35 (7944): No heartbeat from core client for 30 sec - exiting 19:51:36 (7944): No heartbeat from core client for 30 sec - exiting 19:51:37 (7944): No heartbeat from core client for 30 sec - exiting 19:51:38 (7944): No heartbeat from core client for 30 sec - exiting 19:51:39 (7944): No heartbeat from core client for 30 sec - exiting 19:51:40 (7944): No heartbeat from core client for 30 sec - exiting 19:51:41 (7944): No heartbeat from core client for 30 sec - exiting 19:51:42 (7944): No heartbeat from core client for 30 sec - exiting 19:51:43 (7944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:44 (7944): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:09:04 (1892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:32:00 (6876): No heartbeat from core client for 30 sec - exiting 09:32:01 (6876): No heartbeat from core client for 30 sec - exiting 09:32:02 (6876): No heartbeat from core client for 30 sec - exiting 09:32:03 (6876): No heartbeat from core client for 30 sec - exiting 09:32:04 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:51:43 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:44 (3588): No heartbeat from core client for 30 sec - exiting 11:51:45 (3588): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jul 2011 13:15:51	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	233,280	443,364	1.9006
04 Jul 2011 08:38:06	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	207,360	399,624	1.9272
01 Jul 2011 11:49:54	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	181,440	354,117	1.9517
30 Jun 2011 19:57:38	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	155,520	304,317	1.9568
29 Jun 2011 21:40:17	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	129,600	252,780	1.9505
28 Jun 2011 23:59:20	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	103,680	199,555	1.9247
23 Jun 2011 17:18:12	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	77,760	149,279	1.9197
22 Jun 2011 18:36:08	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	51,840	99,629	1.9219
22 Jun 2011 04:03:39	1103150	12993222	hadcm3n_s6q8_1940_40_007301089_1	25,920	49,991	1.9287