Task 16026476

Name	hadcm3n_85j9_1980_40_008465129_0
Workunit	8615968
Created	19 Sep 2013, 14:50:06 UTC
Sent	20 Sep 2013, 1:01:02 UTC
Report deadline	20 Dec 2013, 8:28:13 UTC
Received	16 Oct 2013, 15:37:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1084078
Run time	9 days 11 hours 56 min 20 sec
CPU time	6 days 19 hours 24 min 7 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	1.91 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error 13 - Permission denied Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Skipping gmts_generator due to netcdf error -49 - Variable not found Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6236, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 21:14:39 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:29:46 (10020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4448, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5544, iMonCtr=1 Model crash detected, will try to restart... Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Oct 2013 01:28:35	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	181,440	533,987	2.9431
09 Oct 2013 19:08:04	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	155,520	458,792	2.9501
08 Oct 2013 11:38:24	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	129,600	383,629	2.9601
05 Oct 2013 18:22:21	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	103,680	305,847	2.9499
04 Oct 2013 18:12:02	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	77,760	228,919	2.9439
01 Oct 2013 13:40:10	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	51,840	153,641	2.9638
20 Sep 2013 23:37:43	1084078	16026476	hadcm3n_85j9_1980_40_008465129_0	25,920	76,494	2.9512