Task 13002024

Name	hadcm3n_s4r8_1940_40_007299964_2
Workunit	7497388
Created	24 Jun 2011, 1:47:05 UTC
Sent	24 Jun 2011, 1:47:12 UTC
Report deadline	23 Sep 2011, 9:14:23 UTC
Received	7 Jul 2011, 16:02:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103150
Run time	5 days 4 hours 10 min 32 sec
CPU time	3 days 5 hours 25 min 2 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:26:26 (8508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8092, iMonCtr=1 Model crash detected, will try to restart... 08:35:25 (7148): No heartbeat from core client for 30 sec - exiting 08:35:26 (7148): No heartbeat from core client for 30 sec - exiting 08:35:27 (7148): No heartbeat from core client for 30 sec - exiting 08:35:28 (7148): No heartbeat from core client for 30 sec - exiting 08:35:29 (7148): No heartbeat from core client for 30 sec - exiting 08:35:30 (7148): No heartbeat from core client for 30 sec - exiting 08:35:31 (7148): No heartbeat from core client for 30 sec - exiting 08:35:32 (7148): No heartbeat from core client for 30 sec - exiting 08:35:33 (7148): No heartbeat from core client for 30 sec - exiting 08:35:34 (7148): No heartbeat from core client for 30 sec - exiting 08:35:35 (7148): No heartbeat from core client for 30 sec - exiting 08:35:36 (7148): No heartbeat from core client for 30 sec - exiting 08:35:37 (7148): No heartbeat from core client for 30 sec - exiting 08:35:38 (7148): No heartbeat from core client for 30 sec - exiting 08:35:39 (7148): No heartbeat from core client for 30 sec - exiting 08:35:40 (7148): No heartbeat from core client for 30 sec - exiting 08:35:41 (7148): No heartbeat from core client for 30 sec - exiting 08:35:42 (7148): No heartbeat from core client for 30 sec - exiting 08:35:43 (7148): No heartbeat from core client for 30 sec - exiting 08:35:44 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:45 (7148): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4220, iMonCtr=1 Model crash detected, will try to restart... 09:23:14 (6496): No heartbeat from core client for 30 sec - exiting 09:23:15 (6496): No heartbeat from core client for 30 sec - exiting 09:23:16 (6496): No heartbeat from core client for 30 sec - exiting 09:23:17 (6496): No heartbeat from core client for 30 sec - exiting 09:23:18 (6496): No heartbeat from core client for 30 sec - exiting 09:23:19 (6496): No heartbeat from core client for 30 sec - exiting 09:23:20 (6496): No heartbeat from core client for 30 sec - exiting 09:23:21 (6496): No heartbeat from core client for 30 sec - exiting 09:23:22 (6496): No heartbeat from core client for 30 sec - exiting 09:23:23 (6496): No heartbeat from core client for 30 sec - exiting 09:23:24 (6496): No heartbeat from core client for 30 sec - exiting 09:23:25 (6496): No heartbeat from core client for 30 sec - exiting 09:23:26 (6496): No heartbeat from core client for 30 sec - exiting 09:23:27 (6496): No heartbeat from core client for 30 sec - exiting 09:23:28 (6496): No heartbeat from core client for 30 sec - exiting 09:23:29 (6496): No heartbeat from core client for 30 sec - exiting 09:23:30 (6496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:31 (6496): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:26:17 (9440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:26:19 (9440): No heartbeat from core client for 30 sec - exiting 13:26:20 (9440): No heartbeat from core client for 30 sec - exiting 13:26:21 (9440): No heartbeat from core client for 30 sec - exiting 13:26:22 (9440): No heartbeat from core client for 30 sec - exiting 13:26:23 (9440): No heartbeat from core client for 30 sec - exiting 13:26:24 (9440): No heartbeat from core client for 30 sec - exiting 13:26:25 (9440): No heartbeat from core client for 30 sec - exiting 13:26:26 (9440): No heartbeat from core client for 30 sec - exiting 13:26:27 (9440): No heartbeat from core client for 30 sec - exiting 13:26:28 (9440): No heartbeat from core client for 30 sec - exiting 19:51:29 (7952): No heartbeat from core client for 30 sec - exiting 19:51:30 (7952): No heartbeat from core client for 30 sec - exiting 19:51:31 (7952): No heartbeat from core client for 30 sec - exiting 19:51:32 (7952): No heartbeat from core client for 30 sec - exiting 19:51:33 (7952): No heartbeat from core client for 30 sec - exiting 19:51:34 (7952): No heartbeat from core client for 30 sec - exiting 19:51:35 (7952): No heartbeat from core client for 30 sec - exiting 19:51:36 (7952): No heartbeat from core client for 30 sec - exiting 19:51:37 (7952): No heartbeat from core client for 30 sec - exiting 19:51:38 (7952): No heartbeat from core client for 30 sec - exiting 19:51:39 (7952): No heartbeat from core client for 30 sec - exiting 19:51:40 (7952): No heartbeat from core client for 30 sec - exiting 19:51:41 (7952): No heartbeat from core client for 30 sec - exiting 19:51:42 (7952): No heartbeat from core client for 30 sec - exiting 19:51:43 (7952): No heartbeat from core client for 30 sec - exiting 19:51:44 (7952): No heartbeat from core client for 30 sec - exiting 19:51:45 (7952): No heartbeat from core client for 30 sec - exiting 19:51:46 (7952): No heartbeat from core client for 30 sec - exiting 19:51:47 (7952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:48 (7952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:09:04 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:32:00 (5540): No heartbeat from core client for 30 sec - exiting 09:32:01 (5540): No heartbeat from core client for 30 sec - exiting 09:32:02 (5540): No heartbeat from core client for 30 sec - exiting 09:32:03 (5540): No heartbeat from core client for 30 sec - exiting 09:32:04 (5540): No heartbeat from core client for 30 sec - exiting 09:32:05 (5540): No heartbeat from core client for 30 sec - exiting 09:32:06 (5540): No heartbeat from core client for 30 sec - exiting 09:32:07 (5540): No heartbeat from core client for 30 sec - exiting 09:32:08 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:51:43 (3068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:44 (3068): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2011 16:04:42	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	233,280	420,356	1.8019
07 Jul 2011 16:04:42	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	207,360	378,283	1.8243
07 Jul 2011 16:04:42	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	181,440	335,721	1.8503
05 Jul 2011 16:07:20	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	155,520	294,395	1.8930
04 Jul 2011 10:39:32	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	129,600	249,384	1.9243
01 Jul 2011 06:25:19	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	103,680	202,417	1.9523
30 Jun 2011 14:08:43	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	77,760	152,263	1.9581
29 Jun 2011 15:52:41	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	51,840	100,283	1.9345
28 Jun 2011 19:30:53	1103150	13002024	hadcm3n_s4r8_1940_40_007299964_2	25,920	48,999	1.8904