Task 15807523

Name	hadcm3n_o5zt_2140_40_008269731_4
Workunit	8424855
Created	30 May 2013, 13:52:53 UTC
Sent	30 May 2013, 14:19:45 UTC
Report deadline	29 Aug 2013, 21:46:56 UTC
Received	28 Aug 2013, 3:44:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1238443
Run time	21 days 5 hours 0 min 41 sec
CPU time	17 days 10 hours 39 min 52 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10615, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29234, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:59:43 PM No files match the supplied pattern. MainError: 11:59:43 PM No files match the supplied pattern. Signal 15 received, exiting... Called boinc_finish MainError: 08:13:36 AM No files match the supplied pattern. MainError: 08:13:36 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... MainError: 07:53:47 AM No files match the supplied pattern. MainError: 07:53:47 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... MainError: 06:59:56 PM No files match the supplied pattern. MainError: 06:59:56 PM No files match the supplied pattern. MainError: 02:54:57 AM No files match the supplied pattern. MainError: 02:54:57 AM No files match the supplied pattern. MainError: 10:52:42 AM No files match the supplied pattern. MainError: 10:52:42 AM No files match the supplied pattern. MainError: 06:52:57 PM No files match the supplied pattern. MainError: 06:52:57 PM No files match the supplied pattern. MainError: 02:52:19 AM No files match the supplied pattern. MainError: 02:52:19 AM No files match the supplied pattern. MainError: 10:51:19 AM No files match the supplied pattern. MainError: 10:51:19 AM No files match the supplied pattern. MainError: 06:51:22 PM No files match the supplied pattern. MainError: 06:51:22 PM No files match the supplied pattern. Error converting file to netcdf: dataout/o5ztka.ph11c10 Error converting file to netcdf: dataout/o5ztka.pg11c10 Error converting file to netcdf: dataout/o5ztka.pe11c10 MainError: 02:54:56 AM No files match the supplied pattern. MainError: 02:54:56 AM No files match the supplied pattern. BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Aug 2013 02:56:27	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	777,600	1,701,832	2.1886
27 Aug 2013 18:53:57	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	751,680	1,673,821	2.2268
27 Aug 2013 10:51:53	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	725,760	1,645,985	2.2679
27 Aug 2013 02:54:28	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	699,840	1,618,177	2.3122
26 Aug 2013 18:56:32	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	673,920	1,590,354	2.3599
26 Aug 2013 10:54:52	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	648,000	1,562,529	2.4113
26 Aug 2013 02:57:39	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	622,080	1,534,742	2.4671
25 Aug 2013 19:00:15	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	596,160	1,507,148	2.5281
22 Jun 2013 07:56:07	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	570,240	1,462,688	2.5650
21 Jun 2013 08:18:09	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	544,320	1,379,959	2.5352
20 Jun 2013 00:03:45	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	518,400	1,266,774	2.4436
18 Jun 2013 12:57:41	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	492,480	1,142,261	2.3194
17 Jun 2013 00:32:16	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	466,560	1,012,977	2.1712
15 Jun 2013 15:02:14	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	440,640	893,879	2.0286
14 Jun 2013 21:03:02	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	414,720	829,757	2.0008
14 Jun 2013 04:35:41	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	388,800	771,283	1.9838
13 Jun 2013 12:00:03	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	362,880	712,147	1.9625
12 Jun 2013 19:25:41	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	336,960	653,236	1.9386
12 Jun 2013 02:33:16	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	311,040	593,345	1.9076
11 Jun 2013 09:23:28	1238443	15807523	hadcm3n_o5zt_2140_40_008269731_4	285,120	532,562	1.8679