Task 15473394

Name	hadcm3n_zcuz_1880_40_008251652_2
Workunit	8406776
Created	7 Dec 2012, 8:21:38 UTC
Sent	7 Dec 2012, 8:21:51 UTC
Report deadline	8 Mar 2013, 15:49:02 UTC
Received	19 Dec 2012, 9:24:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1343109
Run time	3 days 18 hours 20 min 21 sec
CPU time	1 hours 11 min 49 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... cpdnmonitor: error closing file dataout/ocean_restart.day forrtl: severe (28): CLOSE error, unit 8, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F761DBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F761DBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: error closing file dataout/ocean_restart.day forrtl: Bad file descriptor forrtl: severe (28): CLOSE error, unit 8, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75E8BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75E8BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75D9BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7583BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7627BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7629BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75DCBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (28): CLOSE error, unit 6, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 084213A0 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F8A6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F754ABD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17638, iMonCtr=1 Model crash detected, will try to restart... Error converting file to netcdf: dataout/zcuzko.pi84c10 forrtl: severe (28): CLOSE error, unit 8, file "Unknown" Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 0841DD18 Unknown Unknown Unknown hadcm3n_um_6.07_i 08121427 Unknown Unknown Unknown hadcm3n_um_6.07_i 083907E5 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7555BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4037, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... OPEN: Unable to Open File dataout/zcuzko.da864i0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2012 02:45:50	1180506	15473394	hadcm3n_zcuz_1880_40_008251652_2	129,600	22,968	0.1772
18 Dec 2012 03:10:43	1180506	15473394	hadcm3n_zcuz_1880_40_008251652_2	103,680	9,793	0.0945
13 Dec 2012 17:35:08	1180506	15473394	hadcm3n_zcuz_1880_40_008251652_2	77,760	14,452	0.1859
13 Dec 2012 17:35:08	1180506	15473394	hadcm3n_zcuz_1880_40_008251652_2	51,840	12,951	0.2498
13 Dec 2012 17:35:08	1180506	15473394	hadcm3n_zcuz_1880_40_008251652_2	25,920	56,268	2.1708