Task 15358115

Name	hadcm3n_o5ea_2100_40_008026097_3
Workunit	8181211
Created	9 Oct 2012, 10:14:14 UTC
Sent	9 Oct 2012, 10:14:20 UTC
Report deadline	8 Jan 2013, 17:41:31 UTC
Received	20 Oct 2012, 5:34:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1213158
Run time	1 days 19 hours 1 min 1 sec
CPU time	1 days 18 hours 45 min 54 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 20:53:17 (8920): No heartbeat from core client for 30 sec - exiting 20:53:18 (8920): No heartbeat from core client for 30 sec - exiting 20:53:19 (8920): No heartbeat from core client for 30 sec - exiting 20:53:20 (8920): No heartbeat from core client for 30 sec - exiting 20:53:21 (8920): No heartbeat from core client for 30 sec - exiting 20:53:22 (8920): No heartbeat from core client for 30 sec - exiting 20:53:23 (8920): No heartbeat from core client for 30 sec - exiting 20:53:24 (8920): No heartbeat from core client for 30 sec - exiting 20:53:25 (8920): No heartbeat from core client for 30 sec - exiting 20:53:26 (8920): No heartbeat from core client for 30 sec - exiting 20:53:27 (8920): No heartbeat from core client for 30 sec - exiting 20:53:28 (8920): No heartbeat from core client for 30 sec - exiting 20:53:29 (8920): No heartbeat from core client for 30 sec - exiting 20:53:30 (8920): No heartbeat from core client for 30 sec - exiting 20:53:31 (8920): No heartbeat from core client for 30 sec - exiting 20:53:32 (8920): No heartbeat from core client for 30 sec - exiting 20:53:33 (8920): No heartbeat from core client for 30 sec - exiting 20:53:34 (8920): No heartbeat from core client for 30 sec - exiting 20:53:35 (8920): No heartbeat from core client for 30 sec - exiting 20:53:36 (8920): No heartbeat from core client for 30 sec - exiting 20:53:37 (8920): No heartbeat from core client for 30 sec - exiting 20:53:38 (8920): No heartbeat from core client for 30 sec - exiting 20:53:39 (8920): No heartbeat from core client for 30 sec - exiting 20:53:40 (8920): No heartbeat from core client for 30 sec - exiting 20:53:41 (8920): No heartbeat from core client for 30 sec - exiting 20:53:42 (8920): No heartbeat from core client for 30 sec - exiting 20:53:43 (8920): No heartbeat from core client for 30 sec - exiting 20:53:44 (8920): No heartbeat from core client for 30 sec - exiting 20:53:45 (8920): No heartbeat from core client for 30 sec - exiting 20:53:46 (8920): No heartbeat from core client for 30 sec - exiting 20:53:47 (8920): No heartbeat from core client for 30 sec - exiting 20:53:48 (8920): No heartbeat from core client for 30 sec - exiting 20:53:49 (8920): No heartbeat from core client for 30 sec - exiting 20:53:50 (8920): No heartbeat from core client for 30 sec - exiting 20:53:51 (8920): No heartbeat from core client for 30 sec - exiting 20:53:52 (8920): No heartbeat from core client for 30 sec - exiting 20:53:53 (8920): No heartbeat from core client for 30 sec - exiting 20:53:54 (8920): No heartbeat from core client for 30 sec - exiting 20:53:55 (8920): No heartbeat from core client for 30 sec - exiting 20:53:56 (8920): No heartbeat from core client for 30 sec - exiting 20:53:57 (8920): No heartbeat from core client for 30 sec - exiting 20:53:58 (8920): No heartbeat from core client for 30 sec - exiting 20:53:59 (8920): No heartbeat from core client for 30 sec - exiting 20:54:00 (8920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:56:00 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:04:46 (8124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o5eako.pju1c10 Error converting file to netcdf: dataout/o5eako.piu1c10 Error converting file to netcdf: dataout/o5eako.pfu1c10 Error converting file to netcdf: dataout/o5eaka.phu1c10 Error converting file to netcdf: dataout/o5eaka.pgu1c10 Error converting file to netcdf: dataout/o5eaka.peu1c10 Error converting file to netcdf: dataout/o5eaka.pdu1c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8920, iMonCtr=1 Model crash detected, will try to restart... 10:47:57 (10260): No heartbeat from core client for 30 sec - exiting 10:47:58 (10260): No heartbeat from core client for 30 sec - exiting 10:47:59 (10260): No heartbeat from core client for 30 sec - exiting 10:48:00 (10260): No heartbeat from core client for 30 sec - exiting 10:48:01 (10260): No heartbeat from core client for 30 sec - exiting 10:48:02 (10260): No heartbeat from core client for 30 sec - exiting 10:48:03 (10260): No heartbeat from core client for 30 sec - exiting 10:48:04 (10260): No heartbeat from core client for 30 sec - exiting 10:48:05 (10260): No heartbeat from core client for 30 sec - exiting 10:48:06 (10260): No heartbeat from core client for 30 sec - exiting 10:48:07 (10260): No heartbeat from core client for 30 sec - exiting 10:48:08 (10260): No heartbeat from core client for 30 sec - exiting 10:48:09 (10260): No heartbeat from core client for 30 sec - exiting 10:48:10 (10260): No heartbeat from core client for 30 sec - exiting 10:48:11 (10260): No heartbeat from core client for 30 sec - exiting 10:48:12 (10260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:13 (10260): No heartbeat from core client for 30 sec - exiting 10:52:09 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:27:08 (7216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:10 (7216): No heartbeat from core client for 30 sec - exiting 10:41:19 (7412): No heartbeat from core client for 30 sec - exiting 10:41:20 (7412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:57 (1488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:56:11 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:47:34 (2924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:47 (6356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:32:10 (10736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:28 (11104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:29 (11104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:53:24 (6640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:02:20 (11068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:02:21 (11068): No heartbeat from core client for 30 sec - exiting 17:02:22 (11068): No heartbeat from core client for 30 sec - exiting 17:02:23 (11068): No heartbeat from core client for 30 sec - exiting 17:02:24 (11068): No heartbeat from core client for 30 sec - exiting 17:02:25 (11068): No heartbeat from core client for 30 sec - exiting 17:25:30 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:01 (10676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:11 (10956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:12 (10956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:16:11 (4560): No heartbeat from core client for 30 sec - exiting 01:16:13 (4560): No heartbeat from core client for 30 sec - exiting 01:16:14 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:23 (12424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:12 (10932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:29 (7296): No heartbeat from core client for 30 sec - exiting 06:53:31 (7296): No heartbeat from core client for 30 sec - exiting 06:53:32 (7296): No heartbeat from core client for 30 sec - exiting 06:53:33 (7296): No heartbeat from core client for 30 sec - exiting 06:53:34 (7296): No heartbeat from core client for 30 sec - exiting 06:53:35 (7296): No heartbeat from core client for 30 sec - exiting 06:53:36 (7296): No heartbeat from core client for 30 sec - exiting 06:53:37 (7296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:12:35 (6132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:52:02 (3992): No heartbeat from core client for 30 sec - exiting 14:52:04 (3992): No heartbeat from core client for 30 sec - exiting 14:52:05 (3992): No heartbeat from core client for 30 sec - exiting 14:52:06 (3992): No heartbeat from core client for 30 sec - exiting 14:52:07 (3992): No heartbeat from core client for 30 sec - exiting 14:52:08 (3992): No heartbeat from core client for 30 sec - exiting 14:52:09 (3992): No heartbeat from core client for 30 sec - exiting 14:52:10 (3992): No heartbeat from core client for 30 sec - exiting 14:52:11 (3992): No heartbeat from core client for 30 sec - exiting 14:52:12 (3992): No heartbeat from core client for 30 sec - exiting 14:52:13 (3992): No heartbeat from core client for 30 sec - exiting 14:52:14 (3992): No heartbeat from core client for 30 sec - exiting 14:52:15 (3992): No heartbeat from core client for 30 sec - exiting 14:52:16 (3992): No heartbeat from core client for 30 sec - exiting 14:52:17 (3992): No heartbeat from core client for 30 sec - exiting 14:52:18 (3992): No heartbeat from core client for 30 sec - exiting 14:52:19 (3992): No heartbeat from core client for 30 sec - exiting 14:52:20 (3992): No heartbeat from core client for 30 sec - exiting 14:52:21 (3992): No heartbeat from core client for 30 sec - exiting 14:52:22 (3992): No heartbeat from core client for 30 sec - exiting 14:52:23 (3992): No heartbeat from core client for 30 sec - exiting 14:52:24 (3992): No heartbeat from core client for 30 sec - exiting 14:52:25 (3992): No heartbeat from core client for 30 sec - exiting 14:52:26 (3992): No heartbeat from core client for 30 sec - exiting 14:52:27 (3992): No heartbeat from core client for 30 sec - exiting 14:52:28 (3992): No heartbeat from core client for 30 sec - exiting 14:52:29 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:52:30 (3992): No heartbeat from core client for 30 sec - exiting 14:54:20 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14872, iMonCtr=1 Model crash detected, will try to restart... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Oct 2012 21:09:23	1213158	15358115	hadcm3n_o5ea_2100_40_008026097_3	103,680	138,813	1.3389
15 Oct 2012 18:49:39	1213158	15358115	hadcm3n_o5ea_2100_40_008026097_3	77,760	104,268	1.3409
14 Oct 2012 18:40:44	1213158	15358115	hadcm3n_o5ea_2100_40_008026097_3	51,840	70,691	1.3636
10 Oct 2012 21:08:04	1213158	15358115	hadcm3n_o5ea_2100_40_008026097_3	25,920	35,375	1.3648