Task 12989424

Name	hadcm3n_s3ir_1940_40_007299478_0
Workunit	7496902
Created	20 Jun 2011, 18:52:13 UTC
Sent	20 Jun 2011, 18:52:29 UTC
Report deadline	20 Sep 2011, 2:19:40 UTC
Received	16 Jul 2011, 13:43:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1063824
Run time	4 days 0 hours 27 min 15 sec
CPU time	3 days 13 hours 8 min 33 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2744, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:49:31 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:19 (3724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2372, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 08:53:03 (3556): No heartbeat from core client for 30 sec - exiting 08:53:05 (3556): No heartbeat from core client for 30 sec - exiting 08:53:06 (3556): No heartbeat from core client for 30 sec - exiting 08:53:07 (3556): No heartbeat from core client for 30 sec - exiting 08:53:08 (3556): No heartbeat from core client for 30 sec - exiting 08:53:09 (3556): No heartbeat from core client for 30 sec - exiting 08:53:10 (3556): No heartbeat from core client for 30 sec - exiting 08:53:11 (3556): No heartbeat from core client for 30 sec - exiting 08:53:12 (3556): No heartbeat from core client for 30 sec - exiting 08:53:13 (3556): No heartbeat from core client for 30 sec - exiting 08:53:14 (3556): No heartbeat from core client for 30 sec - exiting 08:53:15 (3556): No heartbeat from core client for 30 sec - exiting 08:53:17 (3556): No heartbeat from core client for 30 sec - exiting 08:53:18 (3556): No heartbeat from core client for 30 sec - exiting 08:53:19 (3556): No heartbeat from core client for 30 sec - exiting 08:53:20 (3556): No heartbeat from core client for 30 sec - exiting 08:53:21 (3556): No heartbeat from core client for 30 sec - exiting 08:53:22 (3556): No heartbeat from core client for 30 sec - exiting 08:53:23 (3556): No heartbeat from core client for 30 sec - exiting 08:53:24 (3556): No heartbeat from core client for 30 sec - exiting 08:53:25 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:27 (3556): No heartbeat from core client for 30 sec - exiting 15:37:20 (2744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:01:06 (2488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:09 (2488): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5276, selfPID=5276, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:57:58 (2632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:01:41 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:20:23 (2684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:56:10 (2368): No heartbeat from core client for 30 sec - exiting 12:56:11 (2368): No heartbeat from core client for 30 sec - exiting 12:56:12 (2368): No heartbeat from core client for 30 sec - exiting 12:56:13 (2368): No heartbeat from core client for 30 sec - exiting 12:56:14 (2368): No heartbeat from core client for 30 sec - exiting 12:56:15 (2368): No heartbeat from core client for 30 sec - exiting 12:56:16 (2368): No heartbeat from core client for 30 sec - exiting 12:56:18 (2368): No heartbeat from core client for 30 sec - exiting 12:56:19 (2368): No heartbeat from core client for 30 sec - exiting 12:56:20 (2368): No heartbeat from core client for 30 sec - exiting 12:56:21 (2368): No heartbeat from core client for 30 sec - exiting 12:56:22 (2368): No heartbeat from core client for 30 sec - exiting 12:56:23 (2368): No heartbeat from core client for 30 sec - exiting 12:56:24 (2368): No heartbeat from core client for 30 sec - exiting 12:56:25 (2368): No heartbeat from core client for 30 sec - exiting 12:56:26 (2368): No heartbeat from core client for 30 sec - exiting 12:56:27 (2368): No heartbeat from core client for 30 sec - exiting 12:56:28 (2368): No heartbeat from core client for 30 sec - exiting 12:56:30 (2368): No heartbeat from core client for 30 sec - exiting 12:56:31 (2368): No heartbeat from core client for 30 sec - exiting 12:56:32 (2368): No heartbeat from core client for 30 sec - exiting 12:56:33 (2368): No heartbeat from core client for 30 sec - exiting 12:56:34 (2368): No heartbeat from core client for 30 sec - exiting 12:56:35 (2368): No heartbeat from core client for 30 sec - exiting 12:56:36 (2368): No heartbeat from core client for 30 sec - exiting 12:56:37 (2368): No heartbeat from core client for 30 sec - exiting 12:56:38 (2368): No heartbeat from core client for 30 sec - exiting 12:56:39 (2368): No heartbeat from core client for 30 sec - exiting 12:56:40 (2368): No heartbeat from core client for 30 sec - exiting 12:56:42 (2368): No heartbeat from core client for 30 sec - exiting 12:56:43 (2368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:08:12 (5364): No heartbeat from core client for 30 sec - exiting 18:08:13 (5364): No heartbeat from core client for 30 sec - exiting 18:08:14 (5364): No heartbeat from core client for 30 sec - exiting 18:08:15 (5364): No heartbeat from core client for 30 sec - exiting 18:08:16 (5364): No heartbeat from core client for 30 sec - exiting 18:08:17 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:29 (1984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 15:37:01	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	233,280	313,097	1.3422
10 Jul 2011 20:23:39	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	207,360	278,361	1.3424
08 Jul 2011 00:05:42	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	181,440	243,041	1.3395
07 Jul 2011 20:24:01	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	155,520	207,849	1.3365
03 Jul 2011 20:19:43	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	129,600	172,912	1.3342
01 Jul 2011 13:56:26	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	103,680	138,230	1.3332
28 Jun 2011 00:02:55	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	77,760	103,388	1.3296
26 Jun 2011 03:34:50	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	51,840	68,680	1.3248
23 Jun 2011 22:34:58	1063824	12989424	hadcm3n_s3ir_1940_40_007299478_0	25,920	34,568	1.3336