Task 15788013

Name	hadcm3n_39nj_2020_40_008367595_1
Workunit	8518454
Created	17 May 2013, 12:34:47 UTC
Sent	17 May 2013, 12:34:53 UTC
Report deadline	16 Aug 2013, 20:02:04 UTC
Received	23 Jul 2013, 14:18:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1276039
Run time	31 days 8 hours 30 min 34 sec
CPU time	27 days 17 hours 43 min 55 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2644, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 09:26:25 (4976): No heartbeat from core client for 30 sec - exiting 09:26:26 (4976): No heartbeat from core client for 30 sec - exiting 09:26:27 (4976): No heartbeat from core client for 30 sec - exiting 09:26:28 (4976): No heartbeat from core client for 30 sec - exiting 09:26:29 (4976): No heartbeat from core client for 30 sec - exiting 09:26:30 (4976): No heartbeat from core client for 30 sec - exiting 09:26:31 (4976): No heartbeat from core client for 30 sec - exiting 09:26:32 (4976): No heartbeat from core client for 30 sec - exiting 09:26:33 (4976): No heartbeat from core client for 30 sec - exiting 09:26:34 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/39njko.pjm2c10 Error converting file to netcdf: dataout/39njko.pim2c10 Error converting file to netcdf: dataout/39njko.pfm2c10 Error converting file to netcdf: dataout/39njka.phm2c10 Error converting file to netcdf: dataout/39njka.pgm2c10 Error converting file to netcdf: dataout/39njka.pem2c10 Error converting file to netcdf: dataout/39njka.pdm2c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7764, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3780, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:57:44 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:01:54 (1412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:33:21 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:36:11 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=740, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1056, iMonCtr=1 Model crash detected, will try to restart... C13:42:50 (1132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:51 (1132): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4628, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3940, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4048, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5488, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3880, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3896, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3332, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3332, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3852, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3720, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 22:10:22	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	777,600	2,396,623	3.0821
23 Jul 2013 21:00:28	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	751,680	2,318,792	3.0848
23 Jul 2013 20:12:06	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	725,760	2,239,358	3.0855
23 Jul 2013 19:01:54	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	699,840	2,158,480	3.0842
23 Jul 2013 14:34:33	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	673,920	2,071,804	3.0743
23 Jul 2013 14:34:33	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	648,000	1,988,264	3.0683
11 Jul 2013 09:41:24	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	622,080	1,905,684	3.0634
09 Jul 2013 17:03:38	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	596,160	1,824,411	3.0603
06 Jul 2013 05:27:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	570,240	1,741,694	3.0543
02 Jul 2013 20:26:16	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	544,320	1,657,695	3.0454
02 Jul 2013 10:16:15	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	518,400	1,573,450	3.0352
28 Jun 2013 08:28:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	492,480	1,486,301	3.0180
24 Jun 2013 17:24:35	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	466,560	1,403,025	3.0072
21 Jun 2013 17:46:41	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	440,640	1,326,001	3.0093
17 Jun 2013 16:55:40	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	414,720	1,250,036	3.0142
13 Jun 2013 12:35:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	388,800	1,173,464	3.0182
13 Jun 2013 12:35:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	362,880	1,099,990	3.0313
13 Jun 2013 12:35:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	336,960	1,026,829	3.0473
13 Jun 2013 12:35:19	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	311,040	953,121	3.0643
07 Jun 2013 07:50:05	1276039	15788013	hadcm3n_39nj_2020_40_008367595_1	285,120	878,747	3.0820