Task 12642834

Name	famous_vd8e_1599_200_006702916_5
Workunit	6906169
Created	8 Mar 2011, 9:46:37 UTC
Sent	8 Mar 2011, 10:07:10 UTC
Report deadline	7 Jun 2011, 17:34:21 UTC
Received	30 Apr 2011, 5:20:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1131851
Run time	40 days 10 hours 14 min 54 sec
CPU time	24 days 8 hours 11 min 44 sec
Validate state	Invalid
Credit	2,130.92
Device peak FLOPS	0.34 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... (21420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (1619): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (12954): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (29516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... (29544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (16482): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (5210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (8492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (8492): No heartbeat from core client for 30 sec - exiting (8492): No heartbeat from core client for 30 sec - exiting (8492): No heartbeat from core client for 30 sec - exiting (8492): No heartbeat from core client for 30 sec - exiting (8492): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... (14590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14590): No heartbeat from core client for 30 sec - exiting (14590): No heartbeat from core client for 30 sec - exiting (14590): No heartbeat from core client for 30 sec - exiting (14590): No heartbeat from core client for 30 sec - exiting (25531): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_vd8e_1599_200_006702916/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 08267D05 Unknown Unknown Unknown famous_um_6.11_i6 082EFC2C Unknown Unknown Unknown famous_um_6.11_i6 08328714 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 0025CCE7 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8631, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_vd8e_1599_200_006702916/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 08267D05 Unknown Unknown Unknown famous_um_6.11_i6 082EFC2C Unknown Unknown Unknown famous_um_6.11_i6 08328714 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 00126CE7 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8631, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_vd8e_1599_200_006702916/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DF6B0 Unknown Unknown Unknown famous_um_6.11_i6 083DDD57 Unknown Unknown Unknown famous_um_6.11_i6 080B4782 Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8631, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_vd8e_1599_200_006702916/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 08267D05 Unknown Unknown Unknown famous_um_6.11_i6 082EFC2C Unknown Unknown Unknown famous_um_6.11_i6 0817C0B8 Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 00230CE7 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8631, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_vd8e_1599_200_006702916/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DEAA4 Unknown Unknown Unknown famous_um_6.11_i6 08222B6B Unknown Unknown Unknown famous_um_6.11_i6 0824527D Unknown Unknown Unknown famous_um_6.11_i6 08163FDC Unknown Unknown Unknown famous_um_6.11_i6 08163460 Unknown Unknown Unknown famous_um_6.11_i6 081777BF Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 00126CE7 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8631, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( (8631): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2011 22:55:18	1131851	12642834	famous_vd8e_1599_200_006702916_5	645,866	2,078,512	3.2182
20 Apr 2011 15:59:32	1131851	12642834	famous_vd8e_1599_200_006702916_5	636,506	2,048,778	3.2188
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	627,146	2,019,050	3.2194
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	617,786	1,989,381	3.2202
20 Apr 2011 19:10:33	1131851	12642834	famous_vd8e_1599_200_006702916_5	608,426	1,959,641	3.2208
20 Apr 2011 15:59:30	1131851	12642834	famous_vd8e_1599_200_006702916_5	599,066	1,929,981	3.2217
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	589,706	1,900,221	3.2223
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	580,346	1,870,492	3.2231
20 Apr 2011 19:10:33	1131851	12642834	famous_vd8e_1599_200_006702916_5	570,986	1,840,628	3.2236
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	561,626	1,810,725	3.2241
20 Apr 2011 19:10:33	1131851	12642834	famous_vd8e_1599_200_006702916_5	552,266	1,780,843	3.2246
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	542,906	1,751,004	3.2252
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	533,546	1,721,099	3.2258
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	524,186	1,691,150	3.2262
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	514,826	1,661,185	3.2267
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	505,466	1,631,299	3.2273
20 Apr 2011 15:59:33	1131851	12642834	famous_vd8e_1599_200_006702916_5	496,106	1,601,449	3.2280
20 Apr 2011 19:10:35	1131851	12642834	famous_vd8e_1599_200_006702916_5	486,746	1,571,597	3.2288
20 Apr 2011 19:10:34	1131851	12642834	famous_vd8e_1599_200_006702916_5	477,386	1,541,646	3.2293
13 Apr 2011 06:30:23	1131851	12642834	famous_vd8e_1599_200_006702916_5	468,026	1,511,667	3.2299