Task 16231027

Name	hadcm3n_7kky_1980_40_008437957_1
Workunit	8588813
Created	14 Jan 2014, 6:53:56 UTC
Sent	14 Jan 2014, 6:57:27 UTC
Report deadline	15 Apr 2014, 14:24:38 UTC
Received	11 Feb 2014, 5:51:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1309391
Run time	6 days 2 hours 10 min 20 sec
CPU time	5 days 19 hours 45 min 46 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/AvantGarde-Demi.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Helvetica-Oblique.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/NewCenturySchlbk-Roman.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/NewCenturySchlbk-Italic.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Palatino-Roman.txf CreateFile error 32 when trying set file time error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Courier-Oblique.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/AvantGarde-DemiOblique.txf error: cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/txf/Palatino-Bold.txf 10:00:46 (6172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3080, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5724, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3444, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6584, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5400, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:09:59 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6176, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3988, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3988, iMonCtr=1 Model crash detected, will try to restart... 15:23:27 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:01:06 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:19 (6056): No heartbeat from core client for 30 sec - exiting 20:08:20 (6056): No heartbeat from core client for 30 sec - exiting 20:08:21 (6056): No heartbeat from core client for 30 sec - exiting 20:08:23 (6056): No heartbeat from core client for 30 sec - exiting 20:08:24 (6056): No heartbeat from core client for 30 sec - exiting 20:08:25 (6056): No heartbeat from core client for 30 sec - exiting 20:08:26 (6056): No heartbeat from core client for 30 sec - exiting 20:08:27 (6056): No heartbeat from core client for 30 sec - exiting 20:08:28 (6056): No heartbeat from core client for 30 sec - exiting 20:08:29 (6056): No heartbeat from core client for 30 sec - exiting 20:08:30 (6056): No heartbeat from core client for 30 sec - exiting 20:08:31 (6056): No heartbeat from core client for 30 sec - exiting 20:08:32 (6056): No heartbeat from core client for 30 sec - exiting 20:08:33 (6056): No heartbeat from core client for 30 sec - exiting 20:08:34 (6056): No heartbeat from core client for 30 sec - exiting 20:08:35 (6056): No heartbeat from core client for 30 sec - exiting 20:08:36 (6056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6036, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=880, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3048, iMonCtr=1 Model crash detected, will try to restart... Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Model crashed: INITDUMP: Wrong no of ocean prognostic fields tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Feb 2014 16:57:40	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	233,280	471,108	2.0195
06 Feb 2014 04:49:45	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	207,360	419,063	2.0209
03 Feb 2014 07:10:52	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	181,440	367,151	2.0235
31 Jan 2014 17:13:14	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	155,520	316,688	2.0363
30 Jan 2014 16:22:08	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	129,600	266,478	2.0562
27 Jan 2014 17:12:14	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	103,680	215,267	2.0763
25 Jan 2014 17:33:29	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	77,760	164,898	2.1206
23 Jan 2014 06:39:52	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	51,840	114,894	2.2163
19 Jan 2014 16:55:53	1309391	16231027	hadcm3n_7kky_1980_40_008437957_1	25,920	61,933	2.3894