Task 15642438

Name	hadcm3n_4l2u_1940_40_008310212_2
Workunit	8461347
Created	27 Feb 2013, 11:25:09 UTC
Sent	27 Feb 2013, 11:25:17 UTC
Report deadline	29 May 2013, 18:52:28 UTC
Received	3 Apr 2013, 2:35:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1271367
Run time	19 days 12 hours 50 min 29 sec
CPU time	17 days 9 hours 24 min 35 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3488, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2512, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4844, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3468, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4836, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2292, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3848, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4l2uko.pjg8c10 Error converting file to netcdf: dataout/4l2uko.pig8c10 Error converting file to netcdf: dataout/4l2uko.pfg8c10 Error converting file to netcdf: dataout/4l2uka.phg8c10 Error converting file to netcdf: dataout/4l2uka.pgg8c10 Error converting file to netcdf: dataout/4l2uka.peg8c10 Error converting file to netcdf: dataout/4l2uka.pdg8c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2880, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2880, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2916, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Apr 2013 06:52:52	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	777,600	1,499,400	1.9282
01 Apr 2013 00:46:28	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	751,680	1,444,932	1.9223
30 Mar 2013 06:33:43	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	725,760	1,392,301	1.9184
28 Mar 2013 00:05:34	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	699,840	1,338,221	1.9122
27 Mar 2013 07:11:44	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	673,920	1,285,211	1.9071
25 Mar 2013 23:42:34	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	648,000	1,230,782	1.8994
25 Mar 2013 07:16:52	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	622,080	1,178,296	1.8941
23 Mar 2013 00:54:44	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	596,160	1,124,896	1.8869
22 Mar 2013 01:36:47	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	570,240	1,074,267	1.8839
19 Mar 2013 22:24:27	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	544,320	1,020,829	1.8754
19 Mar 2013 03:57:55	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	518,400	968,604	1.8684
17 Mar 2013 02:02:30	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	492,480	915,239	1.8584
15 Mar 2013 23:29:42	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	466,560	862,293	1.8482
15 Mar 2013 07:58:17	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	440,640	810,222	1.8387
13 Mar 2013 22:01:16	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	414,720	755,598	1.8219
12 Mar 2013 02:49:50	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	388,800	702,971	1.8081
10 Mar 2013 15:15:30	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	362,880	649,809	1.7907
09 Mar 2013 03:51:11	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	336,960	599,158	1.7781
06 Mar 2013 23:06:21	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	311,040	547,520	1.7603
06 Mar 2013 04:19:48	1271367	15642438	hadcm3n_4l2u_1940_40_008310212_2	285,120	500,196	1.7543