Task 15695608

Name	hadcm3n_3h5i_1940_40_008258973_3
Workunit	8414097
Created	30 Mar 2013, 13:04:03 UTC
Sent	30 Mar 2013, 13:04:17 UTC
Report deadline	29 Jun 2013, 20:31:28 UTC
Received	23 Apr 2013, 8:42:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1375802
Run time	17 days 15 hours 10 min 51 sec
CPU time	14 days 5 hours 47 min 39 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:51 (21578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:47 (21619): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:05 (21653): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:43 (21810): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:44 (21810): No heartbeat from core client for 30 sec - exiting 02:01:17 (21817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:18 (21817): No heartbeat from core client for 30 sec - exiting 02:01:19 (21817): No heartbeat from core client for 30 sec - exiting 02:01:20 (21817): No heartbeat from core client for 30 sec - exiting 02:02:27 (21826): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:22 (21831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:23 (21831): No heartbeat from core client for 30 sec - exiting 02:04:24 (21831): No heartbeat from core client for 30 sec - exiting 02:05:01 (21838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:08:12 (21846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:08:13 (21846): No heartbeat from core client for 30 sec - exiting 02:08:14 (21846): No heartbeat from core client for 30 sec - exiting 02:08:15 (21846): No heartbeat from core client for 30 sec - exiting 02:08:16 (21846): No heartbeat from core client for 30 sec - exiting 02:08:17 (21846): No heartbeat from core client for 30 sec - exiting 02:08:52 (21858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:38 (21862): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:05:23 (21935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:14:05 (22272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:36:38 (22304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:57:48 (22504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:39 (22593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:30:43 (22602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:24 (22621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:25 (22621): No heartbeat from core client for 30 sec - exiting 01:09:53 (22629): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:49 (22668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:50 (22668): No heartbeat from core client for 30 sec - exiting 03:50:53 (22682): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:50:54 (22682): No heartbeat from core client for 30 sec - exiting 03:50:55 (22682): No heartbeat from core client for 30 sec - exiting 03:50:56 (22682): No heartbeat from core client for 30 sec - exiting 11:25:05 (22737): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:27 (22950): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:28 (22950): No heartbeat from core client for 30 sec - exiting 12:58:16 (22986): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:17 (23014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:33 (23025): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:46:36 (23032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:34 (23077): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:49:09 (23085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:35 (23195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:40:03 (23930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:04 (23930): No heartbeat from core client for 30 sec - exiting 15:40:05 (23930): No heartbeat from core client for 30 sec - exiting 17:01:30 (23961): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:27:18 (3077): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:28:22 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:08 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:09 (3616): No heartbeat from core client for 30 sec - exiting 13:13:21 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:18:54 (3971): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:08:08 (4036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:05:26 (4317): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:39:50 (4514): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/3h5iko.pjf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.pjf0c10 Error: Input file: dataout/3h5iko.pif0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.pif0c10 Error: Input file: dataout/3h5iko.pff0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.pff0c10 Error: Input file: dataout/3h5iko.pcf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.pcf0c10 Error: Input file: dataout/3h5iko.pbf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.pbf0c10 Error: Input file: dataout/3h5iko.paf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5iko.paf0c10 Error: Input file: dataout/3h5ika.phf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5ika.phf0c10 Error: Input file: dataout/3h5ika.pgf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5ika.pgf0c10 Error: Input file: dataout/3h5ika.pef0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5ika.pef0c10 Error: Input file: dataout/3h5ika.pdf0c10 is not a valid UM file. Error converting file to netcdf: dataout/3h5ika.pdf0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:28:21 (4411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:22 (4411): No heartbeat from core client for 30 sec - exiting 18:28:23 (4411): No heartbeat from core client for 30 sec - exiting 18:28:24 (4411): No heartbeat from core client for 30 sec - exiting 18:28:25 (4411): No heartbeat from core client for 30 sec - exiting 18:28:26 (4411): No heartbeat from core client for 30 sec - exiting 18:28:27 (4411): No heartbeat from core client for 30 sec - exiting 18:28:28 (4411): No heartbeat from core client for 30 sec - exiting 18:28:29 (4411): No heartbeat from core client for 30 sec - exiting 18:28:30 (4411): No heartbeat from core client for 30 sec - exiting 18:28:31 (4411): No heartbeat from core client for 30 sec - exiting 18:28:32 (4411): No heartbeat from core client for 30 sec - exiting 18:28:33 (4411): No heartbeat from core client for 30 sec - exiting 18:28:34 (4411): No heartbeat from core client for 30 sec - exiting 18:28:35 (4411): No heartbeat from core client for 30 sec - exiting 18:28:36 (4411): No heartbeat from core client for 30 sec - exiting 18:28:37 (4411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:21:52 (4935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:59 (5201): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:54:00 (5201): No heartbeat from core client for 30 sec - exiting 19:54:01 (5201): No heartbeat from core client for 30 sec - exiting 19:54:02 (5201): No heartbeat from core client for 30 sec - exiting 19:54:03 (5201): No heartbeat from core client for 30 sec - exiting 19:54:04 (5201): No heartbeat from core client for 30 sec - exiting 19:54:05 (5201): No heartbeat from core client for 30 sec - exiting 19:54:06 (5201): No heartbeat from core client for 30 sec - exiting 19:54:07 (5201): No heartbeat from core client for 30 sec - exiting 19:54:08 (5201): No heartbeat from core client for 30 sec - exiting 19:54:09 (5201): No heartbeat from core client for 30 sec - exiting 19:54:10 (5201): No heartbeat from core client for 30 sec - exiting 19:54:11 (5201): No heartbeat from core client for 30 sec - exiting 19:57:05 (5263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:24 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:25 (5280): No heartbeat from core client for 30 sec - exiting 20:00:26 (5280): No heartbeat from core client for 30 sec - exiting 20:00:27 (5280): No heartbeat from core client for 30 sec - exiting 20:00:28 (5280): No heartbeat from core client for 30 sec - exiting 20:00:29 (5280): No heartbeat from core client for 30 sec - exiting 20:00:30 (5280): No heartbeat from core client for 30 sec - exiting 20:00:31 (5280): No heartbeat from core client for 30 sec - exiting 20:00:32 (5280): No heartbeat from core client for 30 sec - exiting 20:00:33 (5280): No heartbeat from core client for 30 sec - exiting 20:00:34 (5280): No heartbeat from core client for 30 sec - exiting 20:00:35 (5280): No heartbeat from core client for 30 sec - exiting 20:00:36 (5280): No heartbeat from core client for 30 sec - exiting 20:00:37 (5280): No heartbeat from core client for 30 sec - exiting 20:00:38 (5280): No heartbeat from core client for 30 sec - exiting 20:00:39 (5280): No heartbeat from core client for 30 sec - exiting 20:00:40 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:41 (5529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:42 (5529): No heartbeat from core client for 30 sec - exiting 22:15:43 (5529): No heartbeat from core client for 30 sec - exiting 22:15:44 (5529): No heartbeat from core client for 30 sec - exiting 22:15:45 (5529): No heartbeat from core client for 30 sec - exiting 22:15:46 (5529): No heartbeat from core client for 30 sec - exiting 22:15:47 (5529): No heartbeat from core client for 30 sec - exiting 22:15:48 (5529): No heartbeat from core client for 30 sec - exiting 22:15:49 (5529): No heartbeat from core client for 30 sec - exiting 22:15:50 (5529): No heartbeat from core client for 30 sec - exiting 22:15:51 (5529): No heartbeat from core client for 30 sec - exiting 22:15:52 (5529): No heartbeat from core client for 30 sec - exiting 22:15:53 (5529): No heartbeat from core client for 30 sec - exiting 22:15:54 (5529): No heartbeat from core client for 30 sec - exiting 22:15:55 (5529): No heartbeat from core client for 30 sec - exiting 22:15:56 (5529): No heartbeat from core client for 30 sec - exiting 22:15:57 (5529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:58:47 (27337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f6400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf76f6425] /lib32/libc.so.6(gsignal+0x4f)[0xf750e34f] /lib32/libc.so.6(abort+0x176)[0xf750fba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74f9923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf771e400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf771e425] /lib32/libc.so.6(gsignal+0x4f)[0xf753634f] /lib32/libc.so.6(abort+0x176)[0xf7537ba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf7521923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779d400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf779d425] /lib32/libc.so.6(gsignal+0x4f)[0xf75b534f] /lib32/libc.so.6(abort+0x176)[0xf75b6ba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75a0923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7776400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf7776425] /lib32/libc.so.6(gsignal+0x4f)[0xf758e34f] /lib32/libc.so.6(abort+0x176)[0xf758fba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf7579923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7742400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf7742425] /lib32/libc.so.6(gsignal+0x4f)[0xf755a34f] /lib32/libc.so.6(abort+0x176)[0xf755bba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf7545923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7773400] linux-gate.so.1(__kernel_vsyscall+0x5)[0xf7773425] /lib32/libc.so.6(gsignal+0x4f)[0xf758b34f] /lib32/libc.so.6(abort+0x176)[0xf758cba6] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/dad/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf7576923] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2484, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Apr 2013 01:31:43	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	829,440	1,214,807	1.4646
21 Apr 2013 19:10:58	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	803,520	1,174,965	1.4623
21 Apr 2013 01:34:57	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	777,600	1,136,518	1.4616
20 Apr 2013 11:19:43	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	751,680	1,096,060	1.4581
19 Apr 2013 23:15:36	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	725,760	1,058,031	1.4578
19 Apr 2013 12:14:50	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	699,840	1,019,718	1.4571
19 Apr 2013 00:05:47	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	673,920	982,309	1.4576
18 Apr 2013 12:02:45	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	648,000	944,740	1.4579
17 Apr 2013 13:28:35	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	622,080	907,679	1.4591
16 Apr 2013 20:21:28	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	596,160	870,654	1.4604
16 Apr 2013 09:34:56	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	570,240	833,461	1.4616
15 Apr 2013 20:27:12	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	544,320	795,976	1.4623
15 Apr 2013 15:19:49	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	518,400	759,584	1.4652
14 Apr 2013 23:32:25	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	492,480	723,103	1.4683
13 Apr 2013 21:37:17	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	466,560	686,462	1.4713
13 Apr 2013 04:21:59	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	440,640	649,074	1.4730
12 Apr 2013 09:29:34	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	414,720	611,649	1.4748
11 Apr 2013 16:51:02	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	388,800	574,329	1.4772
11 Apr 2013 01:51:51	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	362,880	536,482	1.4784
10 Apr 2013 08:43:15	1231967	15695608	hadcm3n_3h5i_1940_40_008258973_3	336,960	496,840	1.4745