Task 14868306

Name	hadcm3n_zkjl_1880_40_008030783_0
Workunit	8185897
Created	5 Jul 2012, 13:44:58 UTC
Sent	5 Jul 2012, 13:45:14 UTC
Report deadline	4 Oct 2012, 21:12:25 UTC
Received	22 Sep 2012, 3:26:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1083033
Run time	42 days 7 hours 5 min 24 sec
CPU time	32 days 20 hours 33 min 8 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	1.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zkjlko.pj83c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3216, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on zkjlko.da90c20 00:43:05 (6660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:43:06 (6660): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5152, iMonCtr=1 Model crash detected, will try to restart... 15:41:27 (4112): No heartbeat from core client for 30 sec - exiting 15:41:28 (4112): No heartbeat from core client for 30 sec - exiting 15:41:29 (4112): No heartbeat from core client for 30 sec - exiting 15:41:31 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:54 (5088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:55 (5088): No heartbeat from core client for 30 sec - exiting 15:30:47 (4764): No heartbeat from core client for 30 sec - exiting 15:30:49 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:50 (4764): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:14:48 (4536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:14 (3576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:32 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:33 (4124): No heartbeat from core client for 30 sec - exiting 22:57:34 (4124): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5040, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4840, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4688, iMonCtr=1 Model crash detected, will try to restart... 03:00:02 (3124): No heartbeat from core client for 30 sec - exiting 03:00:03 (3124): No heartbeat from core client for 30 sec - exiting 03:00:04 (3124): No heartbeat from core client for 30 sec - exiting 03:00:05 (3124): No heartbeat from core client for 30 sec - exiting 03:00:06 (3124): No heartbeat from core client for 30 sec - exiting 03:00:07 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3608, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 22:58:11 (4300): No heartbeat from core client for 30 sec - exiting 22:58:12 (4300): No heartbeat from core client for 30 sec - exiting 22:58:13 (4300): No heartbeat from core client for 30 sec - exiting 22:58:14 (4300): No heartbeat from core client for 30 sec - exiting 22:58:15 (4300): No heartbeat from core client for 30 sec - exiting 22:58:16 (4300): No heartbeat from core client for 30 sec - exiting 22:58:17 (4300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:54 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:37 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:57:07 (3192): No heartbeat from core client for 30 sec - exiting 22:57:08 (3192): No heartbeat from core client for 30 sec - exiting 22:57:09 (3192): No heartbeat from core client for 30 sec - exiting 22:57:10 (3192): No heartbeat from core client for 30 sec - exiting 22:57:11 (3192): No heartbeat from core client for 30 sec - exiting 22:57:12 (3192): No heartbeat from core client for 30 sec - exiting 22:57:13 (3192): No heartbeat from core client for 30 sec - exiting 22:57:14 (3192): No heartbeat from core client for 30 sec - exiting 22:57:15 (3192): No heartbeat from core client for 30 sec - exiting 22:57:16 (3192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:18:17 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:34 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:38:43 (4308): No heartbeat from core client for 30 sec - exiting 01:38:44 (4308): No heartbeat from core client for 30 sec - exiting 01:38:45 (4308): No heartbeat from core client for 30 sec - exiting 01:38:46 (4308): No heartbeat from core client for 30 sec - exiting 01:38:47 (4308): No heartbeat from core client for 30 sec - exiting 01:38:48 (4308): No heartbeat from core client for 30 sec - exiting 01:38:49 (4308): No heartbeat from core client for 30 sec - exiting 01:38:50 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:51 (4308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:56:59 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Sep 2012 11:58:33	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	1,010,880	2,833,023	2.8025
17 Sep 2012 05:49:33	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	984,960	2,760,031	2.8022
10 Sep 2012 19:40:38	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	959,040	2,688,216	2.8030
09 Sep 2012 11:30:37	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	933,120	2,614,461	2.8018
08 Sep 2012 08:56:25	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	907,200	2,542,292	2.8024
30 Aug 2012 09:46:16	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	881,280	2,471,226	2.8041
28 Aug 2012 13:06:58	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	855,360	2,399,807	2.8056
27 Aug 2012 03:30:12	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	829,440	2,328,881	2.8078
26 Aug 2012 04:09:21	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	803,520	2,257,189	2.8091
25 Aug 2012 05:04:00	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	777,600	2,184,496	2.8093
23 Aug 2012 11:17:01	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	751,680	2,113,681	2.8119
21 Aug 2012 16:39:55	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	725,760	2,042,604	2.8144
19 Aug 2012 23:17:33	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	699,840	1,963,657	2.8059
18 Aug 2012 22:01:55	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	673,920	1,884,179	2.7958
17 Aug 2012 08:09:44	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	648,000	1,805,368	2.7861
15 Aug 2012 06:39:35	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	622,080	1,728,198	2.7781
13 Aug 2012 10:09:39	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	596,160	1,655,887	2.7776
11 Aug 2012 11:07:18	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	570,240	1,580,623	2.7719
09 Aug 2012 14:42:22	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	544,320	1,510,607	2.7752
07 Aug 2012 17:23:24	1083033	14868306	hadcm3n_zkjl_1880_40_008030783_0	518,400	1,438,444	2.7748