Task 14291115

Name	hadcm3n_o7ir_1940_40_007833984_1
Workunit	7989096
Created	19 Mar 2012, 14:18:28 UTC
Sent	20 Mar 2012, 0:25:18 UTC
Report deadline	19 Jun 2012, 7:52:29 UTC
Received	18 May 2012, 20:10:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1177198
Run time	17 days 1 hours 57 min 57 sec
CPU time	15 days 15 hours 42 min 5 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.19 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4000, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3492, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o7irko.pje1c10 Error converting file to netcdf: dataout/o7irko.pie1c10 Error converting file to netcdf: dataout/o7irko.pfe1c10 Error converting file to netcdf: dataout/o7irka.phe1c10 Error converting file to netcdf: dataout/o7irka.pge1c10 Error converting file to netcdf: dataout/o7irka.pee1c10 Error converting file to netcdf: dataout/o7irka.pde1c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2632, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3412, iMonCtr=1 Model crash detected, will try to restart... CBUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o7irko.pje4c10 Error converting file to netcdf: dataout/o7irko.pie4c10 Error converting file to netcdf: dataout/o7irko.pfe4c10 Error converting file to netcdf: dataout/o7irka.phe4c10 Error converting file to netcdf: dataout/o7irka.pge4c10 Error converting file to netcdf: dataout/o7irka.pee4c10 Error converting file to netcdf: dataout/o7irka.pde4c10 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:14:13 (3340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3444, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 May 2012 04:26:01	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	777,600	1,352,520	1.7394
17 May 2012 04:29:04	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	751,680	1,312,182	1.7457
16 May 2012 05:03:04	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	725,760	1,268,209	1.7474
15 May 2012 04:38:09	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	699,840	1,223,039	1.7476
14 May 2012 03:31:04	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	673,920	1,177,030	1.7465
13 May 2012 01:28:37	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	648,000	1,131,681	1.7464
11 May 2012 21:47:01	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	622,080	1,086,416	1.7464
10 May 2012 03:34:18	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	596,160	1,040,884	1.7460
08 May 2012 18:22:19	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	570,240	995,875	1.7464
07 May 2012 00:32:36	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	544,320	950,447	1.7461
05 May 2012 02:20:35	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	518,400	904,965	1.7457
02 May 2012 04:05:12	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	492,480	859,356	1.7450
30 Apr 2012 00:03:08	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	466,560	814,146	1.7450
27 Apr 2012 19:12:29	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	440,640	768,442	1.7439
26 Apr 2012 14:29:58	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	414,720	723,298	1.7441
25 Apr 2012 02:55:36	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	388,800	677,916	1.7436
23 Apr 2012 22:43:00	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	362,880	632,887	1.7441
22 Apr 2012 20:41:02	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	336,960	586,380	1.7402
12 Apr 2012 19:16:50	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	311,040	540,406	1.7374
10 Apr 2012 21:07:36	1177198	14291115	hadcm3n_o7ir_1940_40_007833984_1	285,120	494,852	1.7356