Task 15455445

Name	hadcm3n_zcnx_1880_40_008251917_0
Workunit	8407041
Created	22 Nov 2012, 20:18:37 UTC
Sent	22 Nov 2012, 20:30:42 UTC
Report deadline	22 Feb 2013, 3:57:53 UTC
Received	18 Dec 2012, 14:18:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1223462
Run time	9 days 16 hours 13 min 5 sec
CPU time	9 days 7 hours 1 min 29 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	1.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:21:29 (27414): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:30 (27414): No heartbeat from core client for 30 sec - exiting 03:21:31 (27414): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:44:26 (1683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:44:27 (1683): No heartbeat from core client for 30 sec - exiting 10:44:28 (1683): No heartbeat from core client for 30 sec - exiting 10:44:29 (1683): No heartbeat from core client for 30 sec - exiting 10:44:30 (1683): No heartbeat from core client for 30 sec - exiting 10:44:31 (1683): No heartbeat from core client for 30 sec - exiting 10:44:32 (1683): No heartbeat from core client for 30 sec - exiting 10:44:33 (1683): No heartbeat from core client for 30 sec - exiting 10:44:34 (1683): No heartbeat from core client for 30 sec - exiting 10:44:35 (1683): No heartbeat from core client for 30 sec - exiting 10:44:36 (1683): No heartbeat from core client for 30 sec - exiting 10:44:37 (1683): No heartbeat from core client for 30 sec - exiting 10:44:38 (1683): No heartbeat from core client for 30 sec - exiting 10:44:39 (1683): No heartbeat from core client for 30 sec - exiting 10:44:40 (1683): No heartbeat from core client for 30 sec - exiting 10:44:41 (1683): No heartbeat from core client for 30 sec - exiting 10:44:42 (1683): No heartbeat from core client for 30 sec - exiting 10:44:43 (1683): No heartbeat from core client for 30 sec - exiting 10:44:44 (1683): No heartbeat from core client for 30 sec - exiting 10:44:45 (1683): No heartbeat from core client for 30 sec - exiting 10:44:46 (1683): No heartbeat from core client for 30 sec - exiting 10:44:47 (1683): No heartbeat from core client for 30 sec - exiting 10:44:48 (1683): No heartbeat from core client for 30 sec - exiting 10:44:49 (1683): No heartbeat from core client for 30 sec - exiting 10:44:50 (1683): No heartbeat from core client for 30 sec - exiting 10:44:51 (1683): No heartbeat from core client for 30 sec - exiting 10:44:52 (1683): No heartbeat from core client for 30 sec - exiting 10:44:53 (1683): No heartbeat from core client for 30 sec - exiting 10:44:54 (1683): No heartbeat from core client for 30 sec - exiting 10:44:55 (1683): No heartbeat from core client for 30 sec - exiting 10:44:56 (1683): No heartbeat from core client for 30 sec - exiting 10:44:57 (1683): No heartbeat from core client for 30 sec - exiting 10:45:13 (1683): No heartbeat from core client for 30 sec - exiting 10:45:14 (1683): No heartbeat from core client for 30 sec - exiting 10:45:15 (1683): No heartbeat from core client for 30 sec - exiting 10:45:16 (1683): No heartbeat from core client for 30 sec - exiting 10:45:17 (1683): No heartbeat from core client for 30 sec - exiting 10:45:18 (1683): No heartbeat from core client for 30 sec - exiting 10:45:19 (1683): No heartbeat from core client for 30 sec - exiting 10:45:20 (1683): No heartbeat from core client for 30 sec - exiting 10:45:21 (1683): No heartbeat from core client for 30 sec - exiting 10:45:22 (1683): No heartbeat from core client for 30 sec - exiting 10:45:23 (1683): No heartbeat from core client for 30 sec - exiting 10:45:24 (1683): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:58:14 (2002): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (3 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7736400] /lib32/libm.so.6(+0x19824)[0xf76bc824] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77cd400] /lib32/libm.so.6(sin+0x324)[0xf77466c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf758e6b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7731400] /lib32/libm.so.6(sin+0x324)[0xf76aa6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74f26b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b6400] /lib32/libm.so.6(sin+0x324)[0xf772f6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75776b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77ec400] /lib32/libm.so.6(sin+0x324)[0xf77656c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75ad6b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771e400] /lib32/libm.so.6(sin+0x324)[0xf76976c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74df6b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a1400] /lib32/libm.so.6(sin+0x324)[0xf771a6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75626b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7768400] /lib32/libm.so.6(sin+0x324)[0xf76e16c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75296b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Model crashed: SETCONA:Error in locating levels for cloud lyrs tmp/pipe_dummy 2048 SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7715400] /lib32/libm.so.6(sin+0x324)[0xf768e6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74d66b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77e3400] /lib32/libm.so.6(sin+0x324)[0xf775c6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75a46b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Dec 2012 07:34:27	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	596,160	799,066	1.3404
16 Dec 2012 00:22:51	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	570,240	764,787	1.3412
14 Dec 2012 12:10:12	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	544,320	729,919	1.3410
14 Dec 2012 12:10:12	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	518,400	695,592	1.3418
14 Dec 2012 12:10:12	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	492,480	661,180	1.3426
14 Dec 2012 12:10:12	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	466,560	626,707	1.3433
14 Dec 2012 12:10:12	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	440,640	591,242	1.3418
06 Dec 2012 17:52:48	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	414,720	556,078	1.3409
04 Dec 2012 02:45:53	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	388,800	521,278	1.3407
02 Dec 2012 23:12:28	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	362,880	485,689	1.3384
02 Dec 2012 13:19:27	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	336,960	450,588	1.3372
02 Dec 2012 03:22:37	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	311,040	415,542	1.3360
01 Dec 2012 14:57:39	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	285,120	380,172	1.3334
30 Nov 2012 04:20:51	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	259,200	344,233	1.3281
29 Nov 2012 15:08:28	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	233,280	309,821	1.3281
27 Nov 2012 16:45:22	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	207,360	275,484	1.3285
26 Nov 2012 21:40:58	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	181,440	241,171	1.3292
25 Nov 2012 17:17:18	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	155,520	206,749	1.3294
24 Nov 2012 23:03:13	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	129,600	171,963	1.3269
24 Nov 2012 13:19:25	1223462	15455445	hadcm3n_zcnx_1880_40_008251917_0	103,680	137,427	1.3255