Task 13812517

Name	hadcm3n_yd5v_1940_40_007461138_2
Workunit	7658641
Created	23 Dec 2011, 9:39:21 UTC
Sent	23 Dec 2011, 9:47:03 UTC
Report deadline	23 Mar 2012, 17:14:14 UTC
Received	12 Apr 2012, 6:54:10 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1013506
Run time	98 days 18 hours 41 min 38 sec
CPU time	27 days 6 hours 59 min 10 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	1.56 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.6.38</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... 15:37:15 (7516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:32 (7516): No heartbeat from core client for 30 sec - exiting 15:37:33 (7516): No heartbeat from core client for 30 sec - exiting 15:37:34 (7516): No heartbeat from core client for 30 sec - exiting 15:37:35 (7516): No heartbeat from core client for 30 sec - exiting 15:37:36 (7516): No heartbeat from core client for 30 sec - exiting 15:37:37 (7516): No heartbeat from core client for 30 sec - exiting 15:37:38 (7516): No heartbeat from core client for 30 sec - exiting 15:37:39 (7516): No heartbeat from core client for 30 sec - exiting 15:37:40 (7516): No heartbeat from core client for 30 sec - exiting 15:37:45 (7516): No heartbeat from core client for 30 sec - exiting 15:37:46 (7516): No heartbeat from core client for 30 sec - exiting 15:37:47 (7516): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=328, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=328, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1052, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/yd5vko.pje8c10 Error converting file to netcdf: dataout/yd5vko.pie8c10 Error converting file to netcdf: dataout/yd5vko.pfe8c10 Error converting file to netcdf: dataout/yd5vka.phe8c10 Error converting file to netcdf: dataout/yd5vka.pge8c10 Error converting file to netcdf: dataout/yd5vka.pee8c10 Error converting file to netcdf: dataout/yd5vka.pde8c10 15:55:50 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:54 (4408): No heartbeat from core client for 30 sec - exiting 15:55:56 (4408): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2956, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... 22:31:45 (796): No heartbeat from core client for 30 sec - exiting 22:31:46 (796): No heartbeat from core client for 30 sec - exiting 22:31:47 (796): No heartbeat from core client for 30 sec - exiting 22:31:48 (796): No heartbeat from core client for 30 sec - exiting 22:31:49 (796): No heartbeat from core client for 30 sec - exiting 22:31:50 (796): No heartbeat from core client for 30 sec - exiting 22:31:51 (796): No heartbeat from core client for 30 sec - exiting 22:31:52 (796): No heartbeat from core client for 30 sec - exiting 22:31:53 (796): No heartbeat from core client for 30 sec - exiting 22:31:54 (796): No heartbeat from core client for 30 sec - exiting 22:31:55 (796): No heartbeat from core client for 30 sec - exiting 22:31:56 (796): No heartbeat from core client for 30 sec - exiting 22:31:57 (796): No heartbeat from core client for 30 sec - exiting 22:31:58 (796): No heartbeat from core client for 30 sec - exiting 22:31:59 (796): No heartbeat from core client for 30 sec - exiting 22:32:00 (796): No heartbeat from core client for 30 sec - exiting 22:32:01 (796): No heartbeat from core client for 30 sec - exiting 22:32:02 (796): No heartbeat from core client for 30 sec - exiting 22:32:03 (796): No heartbeat from core client for 30 sec - exiting 22:32:04 (796): No heartbeat from core client for 30 sec - exiting 22:32:05 (796): No heartbeat from core client for 30 sec - exiting 22:32:06 (796): No heartbeat from core client for 30 sec - exiting 22:32:07 (796): No heartbeat from core client for 30 sec - exiting 22:32:08 (796): No heartbeat from core client for 30 sec - exiting 22:32:09 (796): No heartbeat from core client for 30 sec - exiting 22:32:10 (796): No heartbeat from core client for 30 sec - exiting 22:32:11 (796): No heartbeat from core client for 30 sec - exiting 22:32:12 (796): No heartbeat from core client for 30 sec - exiting 22:32:13 (796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7704, iMonCtr=1 Model crash detected, will try to restart... 09:29:47 (6140): No heartbeat from core client for 30 sec - exiting 09:29:48 (6140): No heartbeat from core client for 30 sec - exiting 09:29:49 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5808, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6516, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... 20:28:43 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 02:33:59 (7200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:00 (7200): No heartbeat from core client for 30 sec - exiting 02:34:01 (7200): No heartbeat from core client for 30 sec - exiting 02:34:02 (7200): No heartbeat from core client for 30 sec - exiting 02:34:03 (7200): No heartbeat from core client for 30 sec - exiting 02:34:04 (7200): No heartbeat from core client for 30 sec - exiting 02:34:05 (7200): No heartbeat from core client for 30 sec - exiting 02:34:06 (7200): No heartbeat from core client for 30 sec - exiting 02:34:07 (7200): No heartbeat from core client for 30 sec - exiting 02:34:08 (7200): No heartbeat from core client for 30 sec - exiting 02:34:09 (7200): No heartbeat from core client for 30 sec - exiting 21:49:20 (5296): No heartbeat from core client for 30 sec - exiting 21:49:21 (5296): No heartbeat from core client for 30 sec - exiting 21:49:22 (5296): No heartbeat from core client for 30 sec - exiting 21:49:23 (5296): No heartbeat from core client for 30 sec - exiting 21:49:24 (5296): No heartbeat from core client for 30 sec - exiting 21:49:25 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:26 (5296): No heartbeat from core client for 30 sec - exiting 21:49:27 (5296): No heartbeat from core client for 30 sec - exiting 21:49:28 (5296): No heartbeat from core client for 30 sec - exiting 21:49:29 (5296): No heartbeat from core client for 30 sec - exiting 21:49:30 (5296): No heartbeat from core client for 30 sec - exiting 21:49:31 (5296): No heartbeat from core client for 30 sec - exiting 21:49:32 (5296): No heartbeat from core client for 30 sec - exiting 21:49:33 (5296): No heartbeat from core client for 30 sec - exiting 21:49:34 (5296): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5208, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1220, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5252, iMonCtr=1 Model crash detected, will try to restart... 20:57:08 (4184): No heartbeat from core client for 30 sec - exiting 20:57:14 (4184): No heartbeat from core client for 30 sec - exiting 20:57:15 (4184): No heartbeat from core client for 30 sec - exiting 20:57:16 (4184): No heartbeat from core client for 30 sec - exiting 20:57:17 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5248, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6380, iMonCtr=1 Model crash detected, will try to restart... 15:08:19 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:08:26 (4656): No heartbeat from core client for 30 sec - exiting 15:08:27 (4656): No heartbeat from core client for 30 sec - exiting 15:08:28 (4656): No heartbeat from core client for 30 sec - exiting 15:08:29 (4656): No heartbeat from core client for 30 sec - exiting 15:08:30 (4656): No heartbeat from core client for 30 sec - exiting 15:08:31 (4656): No heartbeat from core client for 30 sec - exiting 15:08:32 (4656): No heartbeat from core client for 30 sec - exiting 15:08:33 (4656): No heartbeat from core client for 30 sec - exiting 15:08:34 (4656): No heartbeat from core client for 30 sec - exiting 15:08:35 (4656): No heartbeat from core client for 30 sec - exiting 15:08:36 (4656): No heartbeat from core client for 30 sec - exiting 15:08:37 (4656): No heartbeat from core client for 30 sec - exiting 15:08:38 (4656): No heartbeat from core client for 30 sec - exiting 15:08:39 (4656): No heartbeat from core client for 30 sec - exiting 15:08:40 (4656): No heartbeat from core client for 30 sec - exiting 16:25:53 (4164): No heartbeat from core client for 30 sec - exiting 16:25:54 (4164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:02:31 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:32 (6080): No heartbeat from core client for 30 sec - exiting 03:02:33 (6080): No heartbeat from core client for 30 sec - exiting 03:02:34 (6080): No heartbeat from core client for 30 sec - exiting 03:02:35 (6080): No heartbeat from core client for 30 sec - exiting 03:02:36 (6080): No heartbeat from core client for 30 sec - exiting 03:02:37 (6080): No heartbeat from core client for 30 sec - exiting 03:02:39 (6080): No heartbeat from core client for 30 sec - exiting 03:02:40 (6080): No heartbeat from core client for 30 sec - exiting 03:02:41 (6080): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2076, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2076, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2076, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20544, iMonCtr=1 Model crash detected, will try to restart... 23:08:58 (4320): No heartbeat from core client for 30 sec - exiting 23:08:59 (4320): No heartbeat from core client for 30 sec - exiting 23:09:00 (4320): No heartbeat from core client for 30 sec - exiting 23:09:01 (4320): No heartbeat from core client for 30 sec - exiting 23:09:02 (4320): No heartbeat from core client for 30 sec - exiting 23:09:03 (4320): No heartbeat from core client for 30 sec - exiting 23:09:04 (4320): No heartbeat from core client for 30 sec - exiting 23:09:05 (4320): No heartbeat from core client for 30 sec - exiting 23:09:06 (4320): No heartbeat from core client for 30 sec - exiting 23:09:07 (4320): No heartbeat from core client for 30 sec - exiting 23:09:08 (4320): No heartbeat from core client for 30 sec - exiting 23:09:09 (4320): No heartbeat from core client for 30 sec - exiting 23:09:10 (4320): No heartbeat from core client for 30 sec - exiting 23:09:11 (4320): No heartbeat from core client for 30 sec - exiting 23:09:12 (4320): No heartbeat from core client for 30 sec - exiting 23:09:13 (4320): No heartbeat from core client for 30 sec - exiting 23:09:14 (4320): No heartbeat from core client for 30 sec - exiting 23:09:15 (4320): No heartbeat from core client for 30 sec - exiting 23:09:16 (4320): No heartbeat from core client for 30 sec - exiting 23:09:17 (4320): No heartbeat from core client for 30 sec - exiting 23:09:18 (4320): No heartbeat from core client for 30 sec - exiting 23:09:19 (4320): No heartbeat from core client for 30 sec - exiting 23:09:20 (4320): No heartbeat from core client for 30 sec - exiting 23:09:21 (4320): No heartbeat from core client for 30 sec - exiting 23:09:22 (4320): No heartbeat from core client for 30 sec - exiting 23:09:23 (4320): No heartbeat from core client for 30 sec - exiting 23:09:24 (4320): No heartbeat from core client for 30 sec - exiting 23:09:25 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 03:04:23 (6336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=948, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... 13:41:29 (4884): No heartbeat from core client for 30 sec - exiting 13:41:30 (4884): No heartbeat from core client for 30 sec - exiting 13:41:31 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2828, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 03:09:17 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3444, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 22:20:03 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Apr 2012 05:51:03	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	1,036,800	2,357,145	2.2735
08 Apr 2012 11:52:11	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	1,010,880	2,296,676	2.2720
03 Apr 2012 22:05:42	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	984,960	2,238,492	2.2727
29 Mar 2012 18:09:39	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	959,040	2,178,925	2.2720
24 Mar 2012 18:24:57	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	933,120	2,118,570	2.2704
19 Mar 2012 19:15:32	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	907,200	2,060,176	2.2709
14 Mar 2012 13:14:22	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	881,280	1,996,776	2.2658
08 Mar 2012 16:36:57	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	855,360	1,936,456	2.2639
05 Mar 2012 10:55:45	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	829,440	1,875,959	2.2617
03 Mar 2012 15:52:31	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	803,520	1,815,780	2.2598
02 Mar 2012 19:28:47	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	777,600	1,758,607	2.2616
01 Mar 2012 23:16:07	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	751,680	1,701,227	2.2632
29 Feb 2012 22:32:29	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	725,760	1,648,545	2.2715
27 Feb 2012 19:27:01	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	699,840	1,592,195	2.2751
25 Feb 2012 22:31:01	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	673,920	1,535,587	2.2786
23 Feb 2012 08:09:09	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	648,000	1,475,958	2.2777
19 Feb 2012 22:21:00	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	622,080	1,413,889	2.2728
15 Feb 2012 13:27:41	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	596,160	1,353,063	2.2696
13 Feb 2012 21:38:48	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	570,240	1,296,386	2.2734
12 Feb 2012 21:55:35	1013506	13812517	hadcm3n_yd5v_1940_40_007461138_2	544,320	1,239,759	2.2776