Task 12758897

Name	famous_wnx3_1299_200_007118110_2
Workunit	7316470
Created	31 Mar 2011, 10:43:09 UTC
Sent	31 Mar 2011, 10:43:53 UTC
Report deadline	30 Jun 2011, 18:11:04 UTC
Received	1 Apr 2011, 22:00:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	890423
Run time	19 hours 59 min 33 sec
CPU time	13 hours 14 min 11 sec
Validate state	Invalid
Credit	247.14
Device peak FLOPS	1.53 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy (27817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (27817): No heartbeat from core client for 30 sec - exiting (30906): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (30906): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy (792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (792): No heartbeat from core client for 30 sec - exiting (792): No heartbeat from core client for 30 sec - exiting (792): No heartbeat from core client for 30 sec - exiting (792): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting (20713): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy (27251): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (27251): No heartbeat from core client for 30 sec - exiting (27251): No heartbeat from core client for 30 sec - exiting (10738): No heartbeat from core client for 30 sec - exiting (10738): No heartbeat from core client for 30 sec - exiting (10738): No heartbeat from core client for 30 sec - exiting (10738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (16570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (29584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (31405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (31405): No heartbeat from core client for 30 sec - exiting (6871): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (16439): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (17426): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting (17426): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (22760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (22760): No heartbeat from core client for 30 sec - exiting (22760): No heartbeat from core client for 30 sec - exiting (22760): No heartbeat from core client for 30 sec - exiting (22760): No heartbeat from core client for 30 sec - exiting (22760): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (28499): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (28499): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (12339): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy Sorry, too many model crashes! :-( (17692): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Apr 2011 20:18:28	890423	12758897	famous_wnx3_1299_200_007118110_2	74,906	46,355	0.6188
01 Apr 2011 17:22:32	890423	12758897	famous_wnx3_1299_200_007118110_2	65,546	40,779	0.6221
01 Apr 2011 13:37:55	890423	12758897	famous_wnx3_1299_200_007118110_2	56,186	35,073	0.6242
01 Apr 2011 09:20:14	890423	12758897	famous_wnx3_1299_200_007118110_2	46,826	28,485	0.6083
01 Apr 2011 05:37:29	890423	12758897	famous_wnx3_1299_200_007118110_2	37,466	22,888	0.6109
01 Apr 2011 03:21:28	890423	12758897	famous_wnx3_1299_200_007118110_2	28,106	16,964	0.6036
01 Apr 2011 01:04:59	890423	12758897	famous_wnx3_1299_200_007118110_2	18,746	11,153	0.5950
31 Mar 2011 23:23:41	890423	12758897	famous_wnx3_1299_200_007118110_2	9,386	5,575	0.5940