Task 16295526

Name	hadcm3n_86rm_1980_40_008515273_0
Workunit	8662785
Created	26 Feb 2014, 16:10:04 UTC
Sent	26 Feb 2014, 16:58:23 UTC
Report deadline	29 May 2014, 0:25:34 UTC
Received	17 Apr 2014, 18:44:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1294309
Run time	19 days 9 hours 21 min 18 sec
CPU time	18 days 1 hours 34 min 42 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> Das Gerät erkennt den Befehl nicht. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2472, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2472, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3476, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3764, iMonCtr=1 Model crash detected, will try to restart... 07:34:37 (2572): No heartbeat from core client for 30 sec - exiting 07:34:38 (2572): No heartbeat from core client for 30 sec - exiting 07:34:39 (2572): No heartbeat from core client for 30 sec - exiting 07:34:40 (2572): No heartbeat from core client for 30 sec - exiting 07:34:41 (2572): No heartbeat from core client for 30 sec - exiting 07:34:42 (2572): No heartbeat from core client for 30 sec - exiting 07:34:43 (2572): No heartbeat from core client for 30 sec - exiting 07:34:44 (2572): No heartbeat from core client for 30 sec - exiting 07:34:45 (2572): No heartbeat from core client for 30 sec - exiting 07:34:46 (2572): No heartbeat from core client for 30 sec - exiting 07:34:47 (2572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:35 (2588): No heartbeat from core client for 30 sec - exiting 07:34:36 (2588): No heartbeat from core client for 30 sec - exiting 07:34:37 (2588): No heartbeat from core client for 30 sec - exiting 07:34:38 (2588): No heartbeat from core client for 30 sec - exiting 07:34:39 (2588): No heartbeat from core client for 30 sec - exiting 07:34:40 (2588): No heartbeat from core client for 30 sec - exiting 07:34:41 (2588): No heartbeat from core client for 30 sec - exiting 07:34:42 (2588): No heartbeat from core client for 30 sec - exiting 07:34:43 (2588): No heartbeat from core client for 30 sec - exiting 07:34:44 (2588): No heartbeat from core client for 30 sec - exiting 07:34:45 (2588): No heartbeat from core client for 30 sec - exiting 07:34:46 (2588): No heartbeat from core client for 30 sec - exiting 07:34:47 (2588): No heartbeat from core client for 30 sec - exiting 07:34:48 (2588): No heartbeat from core client for 30 sec - exiting 07:34:49 (2588): No heartbeat from core client for 30 sec - exiting 07:34:50 (2588): No heartbeat from core client for 30 sec - exiting 07:34:51 (2588): No heartbeat from core client for 30 sec - exiting 07:34:52 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=1 Model crash detected, will try to restart... 09:10:29 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5192, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5192, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3688, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3688, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3688, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3700, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1 Model crash detected, will try to restart... 11:00:19 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3980, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3980, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3980, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... 08:00:23 (7928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6832, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 18:15:41 (5408): No heartbeat from core client for 30 sec - exiting 18:15:42 (5408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4940, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4940, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3552, iMonCtr=1 Model crash detected, will try to restart... 16:18:49 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5536, iMonCtr=1 Model crash detected, will try to restart... 17:43:17 (6820): No heartbeat from core client for 30 sec - exiting 17:43:18 (6820): No heartbeat from core client for 30 sec - exiting 17:43:19 (6820): No heartbeat from core client for 30 sec - exiting 17:43:20 (6820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:57:25 (3648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:34:18 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1448, iMonCtr=1 Model crash detected, will try to restart... 20:17:58 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:48 (4656): No heartbeat from core client for 30 sec - exiting 14:23:50 (4656): No heartbeat from core client for 30 sec - exiting 14:23:51 (4656): No heartbeat from core client for 30 sec - exiting 14:23:52 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:30 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:34:02 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4660, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Apr 2014 15:33:43	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	1,010,880	1,537,660	1.5211
12 Apr 2014 08:00:29	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	984,960	1,498,152	1.5210
09 Apr 2014 15:30:16	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	959,040	1,458,695	1.5210
04 Apr 2014 20:02:25	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	933,120	1,419,478	1.5212
03 Apr 2014 17:31:32	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	907,200	1,381,469	1.5228
03 Apr 2014 05:40:10	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	881,280	1,344,467	1.5256
30 Mar 2014 19:43:14	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	855,360	1,305,537	1.5263
29 Mar 2014 14:05:13	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	829,440	1,267,534	1.5282
28 Mar 2014 15:33:17	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	803,520	1,229,659	1.5303
27 Mar 2014 13:09:55	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	777,600	1,191,663	1.5325
26 Mar 2014 14:51:03	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	751,680	1,153,599	1.5347
25 Mar 2014 17:07:18	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	725,760	1,116,058	1.5378
24 Mar 2014 16:08:44	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	699,840	1,076,847	1.5387
23 Mar 2014 16:10:00	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	673,920	1,037,997	1.5402
22 Mar 2014 17:51:32	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	648,000	998,145	1.5403
21 Mar 2014 17:22:05	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	622,080	955,989	1.5368
20 Mar 2014 19:52:59	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	596,160	917,755	1.5394
19 Mar 2014 17:42:57	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	570,240	872,305	1.5297
18 Mar 2014 20:09:56	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	544,320	829,164	1.5233
17 Mar 2014 19:32:03	1294309	16295526	hadcm3n_86rm_1980_40_008515273_0	518,400	790,746	1.5254