Task 11577864

Name	famous_up4d_1999_200_006665400_2
Workunit	6868772
Created	10 Jun 2010, 15:45:25 UTC
Sent	16 Jun 2010, 15:46:33 UTC
Report deadline	15 Sep 2010, 23:13:44 UTC
Received	2 Jul 2010, 18:42:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1039291
Run time
CPU time	1 days 4 hours 2 min 31 sec
Validate state	Invalid
Credit	463.31
Device peak FLOPS	1.73 GFLOPS
Application version	UK Met Office FAMOUS v6.10 i686-pc-linux-gnu
Stderr	<core_client_version>6.4.5</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (1509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1509): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting (1541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (1541): No heartbeat from core client for 30 sec - exiting (2027): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (2027): No heartbeat from core client for 30 sec - exiting (2156): No heartbeat from core client for 30 sec - exiting (2156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (2474): No heartbeat from core client for 30 sec - exiting (2474): No heartbeat from core client for 30 sec - exiting (2474): No heartbeat from core client for 30 sec - exiting (2474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (2474): No heartbeat from core client for 30 sec - exiting (2474): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... (2474): called boinc_finish (1504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (1504): No heartbeat from core client for 30 sec - exiting (1504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (19 frames): /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400] [0xb6a400] /lib/tls/i686/cmov/libm.so.6(expf+0x8)[0xe648a8] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e8f19] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b83c5] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82ed0ea] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e7051] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81e4153] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x827157e] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81299c3] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b668d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x823a5a7] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8077bf6] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8314811] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x832dc4d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8330bdf] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x83415d2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x180b56] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1515, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (19 frames): /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400] [0x440400] /lib/tls/i686/cmov/libm.so.6(expf+0x8)[0x2408a8] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e8f19] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b83c5] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82ed0ea] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e7051] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81e4153] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x827157e] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81299c3] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b668d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x823a5a7] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8077bf6] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8314811] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x832dc4d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8330bdf] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x83415d2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x457b56] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1515, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (19 frames): /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400] [0x5e5400] /lib/tls/i686/cmov/libm.so.6(expf+0x8)[0x2398a8] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e8f19] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b83c5] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82ed0ea] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e7051] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81e4153] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x827157e] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81299c3] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b668d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x823a5a7] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8077bf6] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8314811] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x832dc4d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8330bdf] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x83415d2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x725b56] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1586, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (19 frames): /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400] [0x6d4400] /lib/tls/i686/cmov/libm.so.6(expf+0x8)[0x1948a8] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e8f19] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b83c5] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82ed0ea] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e7051] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81e4153] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x827157e] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81299c3] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b668d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x823a5a7] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8077bf6] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8314811] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x832dc4d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8330bdf] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x83415d2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x353b56] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1586, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (19 frames): /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x83aa400] [0xe6d400] /lib/tls/i686/cmov/libm.so.6(expf+0x8)[0x4068a8] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e8f19] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b83c5] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82ed0ea] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82e7051] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81e4153] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x827157e] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x81299c3] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x82b668d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x823a5a7] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8077bf6] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8314811] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x832dc4d] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x8330bdf] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu[0x83415d2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126b56] /var/lib/boinc-client/projects/climateprediction.net/famous_um_6.10_i686-pc-linux-gnu(pthread_mutex_unlock+0x149)[0x804c041] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1674, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (1674): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2010 16:57:48	1039291	11577864	famous_up4d_1999_200_006665400_2	140,426	96,440	0.6868
01 Jul 2010 14:44:32	1039291	11577864	famous_up4d_1999_200_006665400_2	131,066	89,646	0.6840
29 Jun 2010 19:29:40	1039291	11577864	famous_up4d_1999_200_006665400_2	121,706	83,278	0.6843
29 Jun 2010 17:41:21	1039291	11577864	famous_up4d_1999_200_006665400_2	112,346	76,799	0.6836
26 Jun 2010 18:13:54	1039291	11577864	famous_up4d_1999_200_006665400_2	102,986	70,446	0.6840
24 Jun 2010 07:17:23	1039291	11577864	famous_up4d_1999_200_006665400_2	93,626	63,993	0.6835
22 Jun 2010 20:25:54	1039291	11577864	famous_up4d_1999_200_006665400_2	84,266	57,547	0.6829
22 Jun 2010 18:37:42	1039291	11577864	famous_up4d_1999_200_006665400_2	74,906	51,168	0.6831
22 Jun 2010 16:31:10	1039291	11577864	famous_up4d_1999_200_006665400_2	65,546	44,864	0.6845
22 Jun 2010 09:25:49	1039291	11577864	famous_up4d_1999_200_006665400_2	56,186	38,479	0.6849
21 Jun 2010 19:01:53	1039291	11577864	famous_up4d_1999_200_006665400_2	46,826	32,095	0.6854
21 Jun 2010 14:26:53	1039291	11577864	famous_up4d_1999_200_006665400_2	37,466	25,640	0.6844
19 Jun 2010 08:27:18	1039291	11577864	famous_up4d_1999_200_006665400_2	28,106	19,242	0.6846
18 Jun 2010 14:53:29	1039291	11577864	famous_up4d_1999_200_006665400_2	18,746	12,813	0.6835
17 Jun 2010 17:55:44	1039291	11577864	famous_up4d_1999_200_006665400_2	9,386	6,470	0.6893