Task 11833423

Name	famous_vmbe_1799_200_006714688_4
Workunit	6917941
Created	26 Aug 2010, 17:47:57 UTC
Sent	31 Oct 2010, 10:00:53 UTC
Report deadline	30 Jan 2011, 17:28:04 UTC
Received	20 Nov 2010, 17:13:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	953905
Run time	3 days 11 hours 14 min 27 sec
CPU time	3 days 2 hours 6 min 45 sec
Validate state	Invalid
Credit	1,976.51
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E043D Unknown Unknown Unknown famous_um_6.11_i6 0833E4FA Unknown Unknown Unknown famous_um_6.11_i6 08315CCC Unknown Unknown Unknown famous_um_6.11_i6 08200E9B Unknown Unknown Unknown famous_um_6.11_i6 08173370 Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75F0B96 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E043D Unknown Unknown Unknown famous_um_6.11_i6 0833E4FA Unknown Unknown Unknown famous_um_6.11_i6 08315CCC Unknown Unknown Unknown famous_um_6.11_i6 08200E9B Unknown Unknown Unknown famous_um_6.11_i6 08173370 Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E347A Unknown Unknown Unknown famous_um_6.11_i6 0824B29D Unknown Unknown Unknown famous_um_6.11_i6 08172684 Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75D7B96 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E452A Unknown Unknown Unknown famous_um_6.11_i6 083E3208 Unknown Unknown Unknown famous_um_6.11_i6 08248AA7 Unknown Unknown Unknown famous_um_6.11_i6 081748BE Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F762EB96 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E452A Unknown Unknown Unknown famous_um_6.11_i6 083E3208 Unknown Unknown Unknown famous_um_6.11_i6 08248AA7 Unknown Unknown Unknown famous_um_6.11_i6 081748BE Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F7572B96 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/stssart/BOINC/projects/climateprediction.net/famous_vmbe_1799_200_006714688/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E452A Unknown Unknown Unknown famous_um_6.11_i6 083E3208 Unknown Unknown Unknown famous_um_6.11_i6 08248AA7 Unknown Unknown Unknown famous_um_6.11_i6 081748BE Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F7602B96 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24355, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( (24355): called boinc_finish (18828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (24698): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Nov 2010 15:51:28	953905	11833423	famous_vmbe_1799_200_006714688_4	599,066	264,367	0.4413
20 Nov 2010 13:09:41	953905	11833423	famous_vmbe_1799_200_006714688_4	589,706	260,288	0.4414
20 Nov 2010 11:36:13	953905	11833423	famous_vmbe_1799_200_006714688_4	580,346	256,190	0.4414
20 Nov 2010 10:17:55	953905	11833423	famous_vmbe_1799_200_006714688_4	570,986	252,090	0.4415
20 Nov 2010 08:37:24	953905	11833423	famous_vmbe_1799_200_006714688_4	561,626	248,004	0.4416
20 Nov 2010 06:28:16	953905	11833423	famous_vmbe_1799_200_006714688_4	552,266	243,896	0.4416
20 Nov 2010 06:14:52	953905	11833423	famous_vmbe_1799_200_006714688_4	542,906	239,804	0.4417
20 Nov 2010 06:14:52	953905	11833423	famous_vmbe_1799_200_006714688_4	533,546	235,704	0.4418
20 Nov 2010 06:14:52	953905	11833423	famous_vmbe_1799_200_006714688_4	524,186	231,636	0.4419
20 Nov 2010 05:17:25	953905	11833423	famous_vmbe_1799_200_006714688_4	514,826	227,572	0.4420
19 Nov 2010 22:44:01	953905	11833423	famous_vmbe_1799_200_006714688_4	505,466	223,486	0.4421
19 Nov 2010 20:53:49	953905	11833423	famous_vmbe_1799_200_006714688_4	496,106	219,405	0.4423
19 Nov 2010 19:36:19	953905	11833423	famous_vmbe_1799_200_006714688_4	486,746	215,329	0.4424
19 Nov 2010 18:23:45	953905	11833423	famous_vmbe_1799_200_006714688_4	477,386	211,251	0.4425
19 Nov 2010 17:09:30	953905	11833423	famous_vmbe_1799_200_006714688_4	468,026	207,155	0.4426
19 Nov 2010 15:58:13	953905	11833423	famous_vmbe_1799_200_006714688_4	458,666	203,086	0.4428
19 Nov 2010 14:03:43	953905	11833423	famous_vmbe_1799_200_006714688_4	449,306	199,004	0.4429
19 Nov 2010 11:51:47	953905	11833423	famous_vmbe_1799_200_006714688_4	439,946	194,909	0.4430
19 Nov 2010 10:38:45	953905	11833423	famous_vmbe_1799_200_006714688_4	430,586	190,837	0.4432
19 Nov 2010 09:25:06	953905	11833423	famous_vmbe_1799_200_006714688_4	421,226	186,785	0.4434