Task 15600521

Name	hadcm3n_zbiz_1880_40_008245942_2
Workunit	8401066
Created	8 Feb 2013, 8:52:06 UTC
Sent	8 Feb 2013, 9:00:10 UTC
Report deadline	10 May 2013, 16:27:21 UTC
Received	1 May 2013, 14:43:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1264310
Run time	18 days 11 hours 4 min 23 sec
CPU time	16 days 20 hours 41 min 29 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	2.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:29:04 (5700): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:29:05 (5700): No heartbeat from core client for 30 sec - exiting 20:29:06 (5700): No heartbeat from core client for 30 sec - exiting 20:29:07 (5700): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zbizko.pj95c10 Error converting file to netcdf: dataout/zbizko.pi95c10 Error converting file to netcdf: dataout/zbizko.pf95c10 Error converting file to netcdf: dataout/zbizka.ph95c10 Error converting file to netcdf: dataout/zbizka.pg95c10 Error converting file to netcdf: dataout/zbizka.pe95c10 Error converting file to netcdf: dataout/zbizka.pd95c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:40:13 (5600): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:40:19 (5600): No heartbeat from core client for 30 sec - exiting 15:40:20 (5600): No heartbeat from core client for 30 sec - exiting 15:40:21 (5600): No heartbeat from core client for 30 sec - exiting 15:40:22 (5600): No heartbeat from core client for 30 sec - exiting 15:40:23 (5600): No heartbeat from core client for 30 sec - exiting 15:40:24 (5600): No heartbeat from core client for 30 sec - exiting 15:40:25 (5600): No heartbeat from core client for 30 sec - exiting 15:40:26 (5600): No heartbeat from core client for 30 sec - exiting 15:40:27 (5600): No heartbeat from core client for 30 sec - exiting 15:40:28 (5600): No heartbeat from core client for 30 sec - exiting 15:40:29 (5600): No heartbeat from core client for 30 sec - exiting 15:40:30 (5600): No heartbeat from core client for 30 sec - exiting 15:40:31 (5600): No heartbeat from core client for 30 sec - exiting 15:40:32 (5600): No heartbeat from core client for 30 sec - exiting 15:40:33 (5600): No heartbeat from core client for 30 sec - exiting 15:40:34 (5600): No heartbeat from core client for 30 sec - exiting 15:40:35 (5600): No heartbeat from core client for 30 sec - exiting 15:40:36 (5600): No heartbeat from core client for 30 sec - exiting 15:40:37 (5600): No heartbeat from core client for 30 sec - exiting 15:40:40 (5600): No heartbeat from core client for 30 sec - exiting 15:40:41 (5600): No heartbeat from core client for 30 sec - exiting 15:40:42 (5600): No heartbeat from core client for 30 sec - exiting 15:40:43 (5600): No heartbeat from core client for 30 sec - exiting 15:40:44 (5600): No heartbeat from core client for 30 sec - exiting 15:40:50 (5600): No heartbeat from core client for 30 sec - exiting 15:40:51 (5600): No heartbeat from core client for 30 sec - exiting 15:40:52 (5600): No heartbeat from core client for 30 sec - exiting 15:40:53 (5600): No heartbeat from core client for 30 sec - exiting 15:40:54 (5600): No heartbeat from core client for 30 sec - exiting 15:40:55 (5600): No heartbeat from core client for 30 sec - exiting 15:40:56 (5600): No heartbeat from core client for 30 sec - exiting 15:40:57 (5600): No heartbeat from core client for 30 sec - exiting 15:40:58 (5600): No heartbeat from core client for 30 sec - exiting 15:40:59 (5600): No heartbeat from core client for 30 sec - exiting 15:41:00 (5600): No heartbeat from core client for 30 sec - exiting 15:41:01 (5600): No heartbeat from core client for 30 sec - exiting 15:41:02 (5600): No heartbeat from core client for 30 sec - exiting 15:41:03 (5600): No heartbeat from core client for 30 sec - exiting 15:41:04 (5600): No heartbeat from core client for 30 sec - exiting 15:41:05 (5600): No heartbeat from core client for 30 sec - exiting 15:41:06 (5600): No heartbeat from core client for 30 sec - exiting 15:41:11 (5600): No heartbeat from core client for 30 sec - exiting 15:41:12 (5600): No heartbeat from core client for 30 sec - exiting 15:41:13 (5600): No heartbeat from core client for 30 sec - exiting 15:41:14 (5600): No heartbeat from core client for 30 sec - exiting 15:41:15 (5600): No heartbeat from core client for 30 sec - exiting 15:41:16 (5600): No heartbeat from core client for 30 sec - exiting 15:41:17 (5600): No heartbeat from core client for 30 sec - exiting 15:41:18 (5600): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:51:42 (6952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:51:57 (6952): No heartbeat from core client for 30 sec - exiting 15:51:58 (6952): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2013 12:15:07	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	984,960	1,451,265	1.4734
14 Apr 2013 17:23:07	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	959,040	1,412,030	1.4723
08 Apr 2013 13:40:04	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	933,120	1,374,517	1.4730
07 Apr 2013 11:03:38	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	907,200	1,337,741	1.4746
06 Apr 2013 09:03:45	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	881,280	1,298,245	1.4731
05 Apr 2013 18:37:48	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	855,360	1,260,007	1.4731
05 Apr 2013 09:04:33	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	829,440	1,221,798	1.4730
04 Apr 2013 14:45:32	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	803,520	1,182,654	1.4718
04 Apr 2013 09:00:45	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	777,600	1,143,669	1.4708
03 Apr 2013 10:21:11	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	751,680	1,106,102	1.4715
02 Apr 2013 13:55:22	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	725,760	1,072,226	1.4774
02 Apr 2013 09:13:46	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	699,840	1,038,397	1.4838
01 Apr 2013 12:20:46	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	673,920	1,008,669	1.4967
01 Apr 2013 09:04:29	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	648,000	975,321	1.5051
31 Mar 2013 15:50:44	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	622,080	937,615	1.5072
31 Mar 2013 09:04:12	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	596,160	900,562	1.5106
30 Mar 2013 17:06:00	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	570,240	862,132	1.5119
30 Mar 2013 09:04:27	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	544,320	824,103	1.5140
29 Mar 2013 18:07:17	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	518,400	785,451	1.5151
29 Mar 2013 09:00:59	1264310	15600521	hadcm3n_zbiz_1880_40_008245942_2	492,480	747,366	1.5176