Task 15784549

Name	hadcm3n_3gnl_1980_40_008360894_1
Workunit	8511753
Created	14 May 2013, 22:28:06 UTC
Sent	14 May 2013, 22:28:19 UTC
Report deadline	14 Aug 2013, 5:55:30 UTC
Received	31 May 2013, 21:28:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1265083
Run time	4 days 3 hours 13 min 7 sec
CPU time	3 days 19 hours 22 min 52 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 09:28:03 (2964): No heartbeat from core client for 30 sec - exiting 09:28:04 (2964): No heartbeat from core client for 30 sec - exiting 09:28:05 (2964): No heartbeat from core client for 30 sec - exiting 09:28:06 (2964): No heartbeat from core client for 30 sec - exiting 09:28:07 (2964): No heartbeat from core client for 30 sec - exiting 09:28:08 (2964): No heartbeat from core client for 30 sec - exiting 09:28:09 (2964): No heartbeat from core client for 30 sec - exiting 09:28:10 (2964): No heartbeat from core client for 30 sec - exiting 09:28:12 (2964): No heartbeat from core client for 30 sec - exiting 09:28:13 (2964): No heartbeat from core client for 30 sec - exiting 09:28:14 (2964): No heartbeat from core client for 30 sec - exiting 09:28:15 (2964): No heartbeat from core client for 30 sec - exiting 09:28:16 (2964): No heartbeat from core client for 30 sec - exiting 09:28:17 (2964): No heartbeat from core client for 30 sec - exiting 09:28:18 (2964): No heartbeat from core client for 30 sec - exiting 09:28:19 (2964): No heartbeat from core client for 30 sec - exiting 09:28:20 (2964): No heartbeat from core client for 30 sec - exiting 09:28:21 (2964): No heartbeat from core client for 30 sec - exiting 09:28:22 (2964): No heartbeat from core client for 30 sec - exiting 09:28:24 (2964): No heartbeat from core client for 30 sec - exiting 09:28:25 (2964): No heartbeat from core client for 30 sec - exiting 09:28:26 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4012, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=368, iMonCtr=1 Model crash detected, will try to restart... 09:35:50 (2956): No heartbeat from core client for 30 sec - exiting 09:35:51 (2956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3532, iMonCtr=1 Model crash detected, will try to restart... 09:42:42 (4392): No heartbeat from core client for 30 sec - exiting 09:42:44 (4392): No heartbeat from core client for 30 sec - exiting 09:42:45 (4392): No heartbeat from core client for 30 sec - exiting 09:42:46 (4392): No heartbeat from core client for 30 sec - exiting 09:42:47 (4392): No heartbeat from core client for 30 sec - exiting 09:42:48 (4392): No heartbeat from core client for 30 sec - exiting 09:42:49 (4392): No heartbeat from core client for 30 sec - exiting 09:42:50 (4392): No heartbeat from core client for 30 sec - exiting 09:42:51 (4392): No heartbeat from core client for 30 sec - exiting 09:42:52 (4392): No heartbeat from core client for 30 sec - exiting 09:42:53 (4392): No heartbeat from core client for 30 sec - exiting 09:42:54 (4392): No heartbeat from core client for 30 sec - exiting 09:42:56 (4392): No heartbeat from core client for 30 sec - exiting 09:42:57 (4392): No heartbeat from core client for 30 sec - exiting 09:42:58 (4392): No heartbeat from core client for 30 sec - exiting 09:42:59 (4392): No heartbeat from core client for 30 sec - exiting 09:43:00 (4392): No heartbeat from core client for 30 sec - exiting 09:43:01 (4392): No heartbeat from core client for 30 sec - exiting 09:43:02 (4392): No heartbeat from core client for 30 sec - exiting 09:43:03 (4392): No heartbeat from core client for 30 sec - exiting 09:43:04 (4392): No heartbeat from core client for 30 sec - exiting 09:43:05 (4392): No heartbeat from core client for 30 sec - exiting 09:43:06 (4392): No heartbeat from core client for 30 sec - exiting 09:43:08 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 May 2013 23:57:50	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	259,200	323,014	1.2462
29 May 2013 21:05:18	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	233,280	289,143	1.2395
28 May 2013 19:10:31	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	207,360	256,807	1.2385
24 May 2013 16:15:47	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	181,440	224,522	1.2374
22 May 2013 19:25:30	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	155,520	192,263	1.2363
21 May 2013 17:52:54	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	129,600	161,767	1.2482
20 May 2013 16:29:37	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	103,680	129,687	1.2508
17 May 2013 00:12:03	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	77,760	97,219	1.2502
16 May 2013 06:01:13	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	51,840	65,499	1.2635
15 May 2013 19:45:41	1265083	15784549	hadcm3n_3gnl_1980_40_008360894_1	25,920	33,007	1.2734