Task 11927277

Name	famous_v2ex_799_200_006730561_0
Workunit	6933902
Created	9 Oct 2010, 21:50:56 UTC
Sent	26 Oct 2010, 18:05:05 UTC
Report deadline	26 Jan 2011, 1:32:16 UTC
Received	3 Nov 2010, 14:50:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1109710
Run time	7 days 0 hours 18 min 21 sec
CPU time	6 days 21 hours 28 min 39 sec
Validate state	Invalid
Credit	3,613.24
Device peak FLOPS	2.38 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17401, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17401, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (21911): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Nov 2010 13:59:19	1109710	11927277	famous_v2ex_799_200_006730561_0	1,095,146	592,658	0.5412
03 Nov 2010 12:36:40	1109710	11927277	famous_v2ex_799_200_006730561_0	1,085,786	587,651	0.5412
03 Nov 2010 11:09:32	1109710	11927277	famous_v2ex_799_200_006730561_0	1,076,426	582,632	0.5413
03 Nov 2010 09:46:55	1109710	11927277	famous_v2ex_799_200_006730561_0	1,067,066	577,611	0.5413
03 Nov 2010 08:34:42	1109710	11927277	famous_v2ex_799_200_006730561_0	1,057,706	572,593	0.5414
03 Nov 2010 06:56:35	1109710	11927277	famous_v2ex_799_200_006730561_0	1,048,346	567,377	0.5412
03 Nov 2010 05:30:35	1109710	11927277	famous_v2ex_799_200_006730561_0	1,038,986	562,191	0.5411
03 Nov 2010 04:37:04	1109710	11927277	famous_v2ex_799_200_006730561_0	1,029,626	557,156	0.5411
03 Nov 2010 04:37:04	1109710	11927277	famous_v2ex_799_200_006730561_0	1,020,266	552,119	0.5412
03 Nov 2010 00:13:29	1109710	11927277	famous_v2ex_799_200_006730561_0	1,010,906	547,105	0.5412
02 Nov 2010 22:46:21	1109710	11927277	famous_v2ex_799_200_006730561_0	1,001,546	542,104	0.5413
02 Nov 2010 15:07:53	1109710	11927277	famous_v2ex_799_200_006730561_0	992,186	537,091	0.5413
02 Nov 2010 13:43:43	1109710	11927277	famous_v2ex_799_200_006730561_0	982,826	531,949	0.5412
02 Nov 2010 12:16:28	1109710	11927277	famous_v2ex_799_200_006730561_0	973,466	526,946	0.5413
02 Nov 2010 10:54:24	1109710	11927277	famous_v2ex_799_200_006730561_0	964,106	521,932	0.5414
02 Nov 2010 09:26:14	1109710	11927277	famous_v2ex_799_200_006730561_0	954,746	516,917	0.5414
02 Nov 2010 08:33:58	1109710	11927277	famous_v2ex_799_200_006730561_0	945,386	511,711	0.5413
02 Nov 2010 06:32:34	1109710	11927277	famous_v2ex_799_200_006730561_0	936,026	506,676	0.5413
02 Nov 2010 05:05:41	1109710	11927277	famous_v2ex_799_200_006730561_0	926,666	501,485	0.5412
02 Nov 2010 04:14:57	1109710	11927277	famous_v2ex_799_200_006730561_0	917,306	496,402	0.5412