Task 15902654

Name	hadcm3n_o0ps_2020_40_008403130_0
Workunit	8553986
Created	23 Jul 2013, 11:03:27 UTC
Sent	23 Jul 2013, 13:46:07 UTC
Report deadline	22 Oct 2013, 21:13:18 UTC
Received	29 Jul 2013, 13:41:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1266912
Run time	3 days 17 hours 38 min 42 sec
CPU time	3 days 12 hours 28 min 22 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.43</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 17:42:25 (28913): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:59 (478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:00 (478): No heartbeat from core client for 30 sec - exiting 20:03:01 (478): No heartbeat from core client for 30 sec - exiting 20:03:02 (478): No heartbeat from core client for 30 sec - exiting 20:03:03 (478): No heartbeat from core client for 30 sec - exiting 20:03:04 (478): No heartbeat from core client for 30 sec - exiting 20:03:05 (478): No heartbeat from core client for 30 sec - exiting 20:03:06 (478): No heartbeat from core client for 30 sec - exiting 20:03:07 (478): No heartbeat from core client for 30 sec - exiting 20:03:08 (478): No heartbeat from core client for 30 sec - exiting 20:03:09 (478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:22:44 (30243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:45 (30243): No heartbeat from core client for 30 sec - exiting 07:22:46 (30243): No heartbeat from core client for 30 sec - exiting 07:22:47 (30243): No heartbeat from core client for 30 sec - exiting 07:22:48 (30243): No heartbeat from core client for 30 sec - exiting 07:22:49 (30243): No heartbeat from core client for 30 sec - exiting 07:22:50 (30243): No heartbeat from core client for 30 sec - exiting 07:22:51 (30243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:17:47 (2489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:11:02 (15433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:03 (15433): No heartbeat from core client for 30 sec - exiting 21:11:04 (15433): No heartbeat from core client for 30 sec - exiting 21:11:05 (15433): No heartbeat from core client for 30 sec - exiting 21:11:06 (15433): No heartbeat from core client for 30 sec - exiting 21:11:07 (15433): No heartbeat from core client for 30 sec - exiting 21:11:08 (15433): No heartbeat from core client for 30 sec - exiting 21:11:09 (15433): No heartbeat from core client for 30 sec - exiting 21:11:10 (15433): No heartbeat from core client for 30 sec - exiting 21:11:11 (15433): No heartbeat from core client for 30 sec - exiting 21:11:12 (15433): No heartbeat from core client for 30 sec - exiting 23:01:24 (17197): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:23:38 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:44:16 (4572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:20 (5315): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:21 (5315): No heartbeat from core client for 30 sec - exiting 05:52:22 (5315): No heartbeat from core client for 30 sec - exiting 05:52:23 (5315): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:18:10 (9464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:29:23 (13853): No heartbeat from core client for 30 sec - exiting 09:29:24 (13853): No heartbeat from core client for 30 sec - exiting 09:29:25 (13853): No heartbeat from core client for 30 sec - exiting 09:29:26 (13853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o0psko.pjm2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0psko.pjm2c10 Error: Input file: dataout/o0psko.pim2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0psko.pim2c10 Error: Input file: dataout/o0psko.pfm2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0psko.pfm2c10 Error: Input file: dataout/o0pska.phm2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0pska.phm2c10 Error: Input file: dataout/o0pska.pgm2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0pska.pgm2c10 Error: Input file: dataout/o0pska.pem2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0pska.pem2c10 Error: Input file: dataout/o0pska.pdm2c10 is not a valid UM file. Error converting file to netcdf: dataout/o0pska.pdm2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:41:47 (1278): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:42 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:43 (5620): No heartbeat from core client for 30 sec - exiting 21:23:35 (7433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:07 (9722): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:34:58 (12827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:07:00 (26419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:01 (26419): No heartbeat from core client for 30 sec - exiting 04:07:02 (26419): No heartbeat from core client for 30 sec - exiting 04:07:03 (26419): No heartbeat from core client for 30 sec - exiting 04:07:04 (26419): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 06:18:26 (30383): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:33:35 (5433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:33:36 (5433): No heartbeat from core client for 30 sec - exiting 09:56:12 (7511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:13 (7511): No heartbeat from core client for 30 sec - exiting 09:56:14 (7511): No heartbeat from core client for 30 sec - exiting 09:56:15 (7511): No heartbeat from core client for 30 sec - exiting 09:56:16 (7511): No heartbeat from core client for 30 sec - exiting 09:56:17 (7511): No heartbeat from core client for 30 sec - exiting 09:56:18 (7511): No heartbeat from core client for 30 sec - exiting 09:56:19 (7511): No heartbeat from core client for 30 sec - exiting 09:56:20 (7511): No heartbeat from core client for 30 sec - exiting 09:56:21 (7511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:59:53 (22717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:41 (24889): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:30:47 (27641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:49 (27641): No heartbeat from core client for 30 sec - exiting 21:30:50 (27641): No heartbeat from core client for 30 sec - exiting 21:30:51 (27641): No heartbeat from core client for 30 sec - exiting 21:30:52 (27641): No heartbeat from core client for 30 sec - exiting 21:41:10 (28524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:41:11 (28524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:14:05 (2602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:06 (3527): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:07 (3527): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:18:27 (6444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:36 (7756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:13 (8730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:14 (8730): No heartbeat from core client for 30 sec - exiting 09:12:15 (8730): No heartbeat from core client for 30 sec - exiting 09:12:16 (8730): No heartbeat from core client for 30 sec - exiting 09:54:09 (9654): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:07:52 (10357): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:07:53 (10357): No heartbeat from core client for 30 sec - exiting 12:07:54 (10357): No heartbeat from core client for 30 sec - exiting 12:07:55 (10357): No heartbeat from core client for 30 sec - exiting 12:07:56 (10357): No heartbeat from core client for 30 sec - exiting 12:07:57 (10357): No heartbeat from core client for 30 sec - exiting 12:07:58 (10357): No heartbeat from core client for 30 sec - exiting 12:07:59 (10357): No heartbeat from core client for 30 sec - exiting 12:08:00 (10357): No heartbeat from core client for 30 sec - exiting 12:08:01 (10357): No heartbeat from core client for 30 sec - exiting 12:08:02 (10357): No heartbeat from core client for 30 sec - exiting 12:08:03 (10357): No heartbeat from core client for 30 sec - exiting 12:08:04 (10357): No heartbeat from core client for 30 sec - exiting 12:08:05 (10357): No heartbeat from core client for 30 sec - exiting 13:18:21 (12683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:22 (12683): No heartbeat from core client for 30 sec - exiting 13:18:23 (12683): No heartbeat from core client for 30 sec - exiting 18:49:04 (13868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:27 (19423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:28 (19423): No heartbeat from core client for 30 sec - exiting 20:17:03 (20246): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:04 (20246): No heartbeat from core client for 30 sec - exiting 20:17:05 (20246): No heartbeat from core client for 30 sec - exiting 20:17:06 (20246): No heartbeat from core client for 30 sec - exiting 20:34:19 (21069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:20 (21069): No heartbeat from core client for 30 sec - exiting 21:42:48 (21361): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:16 (22595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:26:56 (25045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:26:57 (25045): No heartbeat from core client for 30 sec - exiting 00:26:58 (25045): No heartbeat from core client for 30 sec - exiting 00:26:59 (25045): No heartbeat from core client for 30 sec - exiting 00:27:00 (25045): No heartbeat from core client for 30 sec - exiting 00:27:01 (25045): No heartbeat from core client for 30 sec - exiting 00:27:02 (25045): No heartbeat from core client for 30 sec - exiting 00:27:03 (25045): No heartbeat from core client for 30 sec - exiting 00:27:04 (25045): No heartbeat from core client for 30 sec - exiting 01:17:52 (26214): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:10 (27518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:51:00 (30937): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:01 (30937): No heartbeat from core client for 30 sec - exiting 04:51:02 (30937): No heartbeat from core client for 30 sec - exiting 04:51:03 (30937): No heartbeat from core client for 30 sec - exiting 05:05:10 (31472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 06:53:44 (479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:45:26 (6181): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:27 (6181): No heartbeat from core client for 30 sec - exiting 12:45:28 (6181): No heartbeat from core client for 30 sec - exiting 13:56:45 (7713): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:53:22 (10907): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:37 (11121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:13 (12934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:40 (13520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:41 (13520): No heartbeat from core client for 30 sec - exiting 19:04:42 (13520): No heartbeat from core client for 30 sec - exiting 19:04:43 (13520): No heartbeat from core client for 30 sec - exiting 19:04:44 (13520): No heartbeat from core client for 30 sec - exiting 19:04:45 (13520): No heartbeat from core client for 30 sec - exiting 19:04:46 (13520): No heartbeat from core client for 30 sec - exiting 19:04:47 (13520): No heartbeat from core client for 30 sec - exiting 19:04:48 (13520): No heartbeat from core client for 30 sec - exiting 19:04:49 (13520): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f3400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76f3430] /lib/libc.so.6(gsignal+0x4f)[0xf752d31f] /lib/libc.so.6(abort+0x143)[0xf752ec03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75183d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7781400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7781430] /lib/libc.so.6(gsignal+0x4f)[0xf75bb31f] /lib/libc.so.6(abort+0x143)[0xf75bcc03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75a63d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7724400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7724430] /lib/libc.so.6(gsignal+0x4f)[0xf755e31f] /lib/libc.so.6(abort+0x143)[0xf755fc03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75493d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7786400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7786430] /lib/libc.so.6(gsignal+0x4f)[0xf75c031f] /lib/libc.so.6(abort+0x143)[0xf75c1c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75ab3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779b400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf779b430] /lib/libc.so.6(gsignal+0x4f)[0xf75d531f] /lib/libc.so.6(abort+0x143)[0xf75d6c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75c03d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7756400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7756430] /lib/libc.so.6(gsignal+0x4f)[0xf759031f] /lib/libc.so.6(abort+0x143)[0xf7591c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf757b3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14653, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jul 2013 13:52:48	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	155,520	301,658	1.9397
29 Jul 2013 13:52:48	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	129,600	251,997	1.9444
29 Jul 2013 13:52:48	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	103,680	201,728	1.9457
26 Jul 2013 04:56:03	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	77,760	151,039	1.9424
25 Jul 2013 07:29:28	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	51,840	100,643	1.9414
24 Jul 2013 11:34:25	1266912	15902654	hadcm3n_o0ps_2020_40_008403130_0	25,920	51,503	1.9870