Task 15812282

Name	hadcm3n_o6vl_1940_40_008381088_0
Workunit	8531947
Created	1 Jun 2013, 0:08:18 UTC
Sent	15 Jun 2013, 15:24:38 UTC
Report deadline	14 Sep 2013, 22:51:49 UTC
Received	19 Jun 2013, 22:37:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1266912
Run time	2 days 5 hours 21 min 9 sec
CPU time	2 days 1 hours 42 min 34 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.43</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 18:17:41 (4981): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:28 (5551): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:29 (5551): No heartbeat from core client for 30 sec - exiting 18:47:30 (5551): No heartbeat from core client for 30 sec - exiting 18:47:31 (5551): No heartbeat from core client for 30 sec - exiting 19:58:41 (6685): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:03:14 (15481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:41 (16621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:32:11 (18101): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:46:50 (25672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:01 (25672): No heartbeat from core client for 30 sec - exiting 02:47:02 (25672): No heartbeat from core client for 30 sec - exiting 03:38:41 (26144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:42 (26144): No heartbeat from core client for 30 sec - exiting 04:37:45 (28177): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:37:01 (2683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:57:46 (7845): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:57 (7845): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:25:41 (23442): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:05:25 (28373): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:26 (28373): No heartbeat from core client for 30 sec - exiting 18:05:27 (28373): No heartbeat from core client for 30 sec - exiting 18:10:25 (29138): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:59:49 (13544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:52 (13544): No heartbeat from core client for 30 sec - exiting 01:23:53 (14323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:23:54 (14323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 03:43:46 (19972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:56:10 (27154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:59 (27226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:00 (27226): No heartbeat from core client for 30 sec - exiting 08:01:01 (27226): No heartbeat from core client for 30 sec - exiting 08:01:02 (27226): No heartbeat from core client for 30 sec - exiting 08:01:03 (27226): No heartbeat from core client for 30 sec - exiting 08:01:04 (27226): No heartbeat from core client for 30 sec - exiting 08:01:05 (27226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:28:59 (31557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:34 (31953): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:23:07 (1469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:15:51 (3147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:29:25 (13323): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:05:21 (31223): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:51:41 (7537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:01 (9396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:12:45 (15667): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:12:46 (15667): No heartbeat from core client for 30 sec - exiting 06:12:47 (15667): No heartbeat from core client for 30 sec - exiting 06:46:04 (17166): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:46:05 (17166): No heartbeat from core client for 30 sec - exiting 07:12:03 (18416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:04 (18416): No heartbeat from core client for 30 sec - exiting 07:12:05 (18416): No heartbeat from core client for 30 sec - exiting 07:12:06 (18416): No heartbeat from core client for 30 sec - exiting 07:12:07 (18416): No heartbeat from core client for 30 sec - exiting 07:12:08 (18416): No heartbeat from core client for 30 sec - exiting 07:12:09 (18416): No heartbeat from core client for 30 sec - exiting 07:58:23 (19444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:51 (21218): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:52 (21218): No heartbeat from core client for 30 sec - exiting 08:28:53 (21218): No heartbeat from core client for 30 sec - exiting 08:28:54 (21218): No heartbeat from core client for 30 sec - exiting 08:28:55 (21218): No heartbeat from core client for 30 sec - exiting 08:28:56 (21218): No heartbeat from core client for 30 sec - exiting 08:28:57 (21218): No heartbeat from core client for 30 sec - exiting 08:28:58 (21218): No heartbeat from core client for 30 sec - exiting 08:28:59 (21218): No heartbeat from core client for 30 sec - exiting 08:29:00 (21218): No heartbeat from core client for 30 sec - exiting 08:29:01 (21218): No heartbeat from core client for 30 sec - exiting 08:29:02 (21218): No heartbeat from core client for 30 sec - exiting 08:29:03 (21218): No heartbeat from core client for 30 sec - exiting 08:29:04 (21218): No heartbeat from core client for 30 sec - exiting 08:29:05 (21218): No heartbeat from core client for 30 sec - exiting 08:36:58 (22390): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:32 (22711): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:41 (24892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:04 (25047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:47:31 (28529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:27:46 (7958): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:43 (10533): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:00 (10533): No heartbeat from core client for 30 sec - exiting 19:53:01 (10533): No heartbeat from core client for 30 sec - exiting 20:27:10 (17173): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:27:11 (17173): No heartbeat from core client for 30 sec - exiting 20:27:12 (17173): No heartbeat from core client for 30 sec - exiting 20:27:13 (17173): No heartbeat from core client for 30 sec - exiting 20:27:14 (17173): No heartbeat from core client for 30 sec - exiting 20:27:15 (17173): No heartbeat from core client for 30 sec - exiting 20:27:16 (17173): No heartbeat from core client for 30 sec - exiting 20:27:17 (17173): No heartbeat from core client for 30 sec - exiting 20:27:18 (17173): No heartbeat from core client for 30 sec - exiting 20:27:19 (17173): No heartbeat from core client for 30 sec - exiting 20:27:20 (17173): No heartbeat from core client for 30 sec - exiting 20:27:21 (17173): No heartbeat from core client for 30 sec - exiting 20:27:22 (17173): No heartbeat from core client for 30 sec - exiting 20:27:23 (17173): No heartbeat from core client for 30 sec - exiting 20:27:24 (17173): No heartbeat from core client for 30 sec - exiting 20:27:25 (17173): No heartbeat from core client for 30 sec - exiting 20:27:26 (17173): No heartbeat from core client for 30 sec - exiting 20:27:27 (17173): No heartbeat from core client for 30 sec - exiting 20:27:28 (17173): No heartbeat from core client for 30 sec - exiting 20:27:29 (17173): No heartbeat from core client for 30 sec - exiting 20:27:30 (17173): No heartbeat from core client for 30 sec - exiting 20:27:31 (17173): No heartbeat from core client for 30 sec - exiting 20:27:32 (17173): No heartbeat from core client for 30 sec - exiting 20:27:33 (17173): No heartbeat from core client for 30 sec - exiting 20:27:34 (17173): No heartbeat from core client for 30 sec - exiting 20:27:35 (17173): No heartbeat from core client for 30 sec - exiting 20:27:36 (17173): No heartbeat from core client for 30 sec - exiting 20:37:12 (17284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:13 (17284): No heartbeat from core client for 30 sec - exiting 20:37:14 (17284): No heartbeat from core client for 30 sec - exiting 20:37:15 (17284): No heartbeat from core client for 30 sec - exiting 20:37:16 (17284): No heartbeat from core client for 30 sec - exiting 20:37:17 (17284): No heartbeat from core client for 30 sec - exiting 20:37:18 (17284): No heartbeat from core client for 30 sec - exiting 20:37:19 (17284): No heartbeat from core client for 30 sec - exiting 20:37:20 (17284): No heartbeat from core client for 30 sec - exiting 20:37:21 (17284): No heartbeat from core client for 30 sec - exiting 20:40:10 (17660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:11 (17660): No heartbeat from core client for 30 sec - exiting 20:40:12 (17660): No heartbeat from core client for 30 sec - exiting 20:40:13 (17660): No heartbeat from core client for 30 sec - exiting 20:40:14 (17660): No heartbeat from core client for 30 sec - exiting 20:40:15 (17660): No heartbeat from core client for 30 sec - exiting 20:40:16 (17660): No heartbeat from core client for 30 sec - exiting 20:40:17 (17660): No heartbeat from core client for 30 sec - exiting 20:40:18 (17660): No heartbeat from core client for 30 sec - exiting 20:42:42 (17818): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:10 (17884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:44 (18042): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:05 (18136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:52 (18275): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:53 (18275): No heartbeat from core client for 30 sec - exiting 20:57:54 (18275): No heartbeat from core client for 30 sec - exiting 20:57:55 (18275): No heartbeat from core client for 30 sec - exiting 20:57:56 (18275): No heartbeat from core client for 30 sec - exiting 20:57:57 (18275): No heartbeat from core client for 30 sec - exiting 20:57:58 (18275): No heartbeat from core client for 30 sec - exiting 20:57:59 (18275): No heartbeat from core client for 30 sec - exiting 20:58:00 (18275): No heartbeat from core client for 30 sec - exiting 20:58:01 (18275): No heartbeat from core client for 30 sec - exiting 20:58:02 (18275): No heartbeat from core client for 30 sec - exiting 20:58:03 (18275): No heartbeat from core client for 30 sec - exiting 21:00:42 (18508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:02 (18683): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:03 (18683): No heartbeat from core client for 30 sec - exiting 21:07:04 (18683): No heartbeat from core client for 30 sec - exiting 21:07:05 (18683): No heartbeat from core client for 30 sec - exiting 21:07:06 (18683): No heartbeat from core client for 30 sec - exiting 21:07:07 (18683): No heartbeat from core client for 30 sec - exiting 21:07:08 (18683): No heartbeat from core client for 30 sec - exiting 21:43:25 (18842): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:29 (18842): No heartbeat from core client for 30 sec - exiting 21:43:30 (18842): No heartbeat from core client for 30 sec - exiting 21:44:35 (20210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:36 (20210): No heartbeat from core client for 30 sec - exiting 21:51:22 (20277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:51:23 (20277): No heartbeat from core client for 30 sec - exiting 21:51:24 (20277): No heartbeat from core client for 30 sec - exiting 21:51:25 (20277): No heartbeat from core client for 30 sec - exiting 21:57:29 (20545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:30 (20545): No heartbeat from core client for 30 sec - exiting 22:01:46 (20771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:01:47 (20771): No heartbeat from core client for 30 sec - exiting 22:01:48 (20771): No heartbeat from core client for 30 sec - exiting 22:01:49 (20771): No heartbeat from core client for 30 sec - exiting 22:15:05 (21052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:06 (21052): No heartbeat from core client for 30 sec - exiting 22:15:07 (21052): No heartbeat from core client for 30 sec - exiting 22:17:22 (21534): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:58 (21574): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:52:21 (21981): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:15 (22888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:23 (24497): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:24 (24497): No heartbeat from core client for 30 sec - exiting 00:12:25 (24497): No heartbeat from core client for 30 sec - exiting 00:12:26 (24497): No heartbeat from core client for 30 sec - exiting 00:12:27 (24497): No heartbeat from core client for 30 sec - exiting 00:12:28 (24497): No heartbeat from core client for 30 sec - exiting 00:12:29 (24497): No heartbeat from core client for 30 sec - exiting 00:12:30 (24497): No heartbeat from core client for 30 sec - exiting 00:12:31 (24497): No heartbeat from core client for 30 sec - exiting 00:12:32 (24497): No heartbeat from core client for 30 sec - exiting 00:12:33 (24497): No heartbeat from core client for 30 sec - exiting 00:12:34 (24497): No heartbeat from core client for 30 sec - exiting 00:12:35 (24497): No heartbeat from core client for 30 sec - exiting 00:12:36 (24497): No heartbeat from core client for 30 sec - exiting 00:12:37 (24497): No heartbeat from core client for 30 sec - exiting 00:24:53 (26097): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:47:39 (26584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:42 (27863): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:57 (28271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:26:40 (28491): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:26:41 (28491): No heartbeat from core client for 30 sec - exiting 01:26:42 (28491): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf778d400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf778d430] /lib/libc.so.6(gsignal+0x4f)[0xf75c731f] /lib/libc.so.6(abort+0x143)[0xf75c8c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75b23d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7707400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7707430] /lib/libc.so.6(gsignal+0x4f)[0xf754131f] /lib/libc.so.6(abort+0x143)[0xf7542c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf752c3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7774400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7774430] /lib/libc.so.6(gsignal+0x4f)[0xf75ae31f] /lib/libc.so.6(abort+0x143)[0xf75afc03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75993d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7745400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7745430] /lib/libc.so.6(gsignal+0x4f)[0xf757f31f] /lib/libc.so.6(abort+0x143)[0xf7580c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf756a3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7787400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf7787430] /lib/libc.so.6(gsignal+0x4f)[0xf75c131f] /lib/libc.so.6(abort+0x143)[0xf75c2c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf75ac3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf76f7400] linux-gate.so.1(__kernel_vsyscall+0x10)[0xf76f7430] /lib/libc.so.6(gsignal+0x4f)[0xf753131f] /lib/libc.so.6(abort+0x143)[0xf7532c03] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xf5)[0xf751c3d5] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29307, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Jun 2013 06:07:37	1266912	15812282	hadcm3n_o6vl_1940_40_008381088_0	77,760	139,063	1.7884
17 Jun 2013 10:01:32	1266912	15812282	hadcm3n_o6vl_1940_40_008381088_0	51,840	89,655	1.7295
16 Jun 2013 13:12:44	1266912	15812282	hadcm3n_o6vl_1940_40_008381088_0	25,920	45,564	1.7579