Task 13661893

Name	hadcm3n_ylrh_1900_40_007513844_3
Workunit	7711319
Created	25 Nov 2011, 19:53:36 UTC
Sent	25 Nov 2011, 20:13:26 UTC
Report deadline	25 Feb 2012, 3:40:37 UTC
Received	11 Dec 2011, 10:31:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181044
Run time	4 days 20 hours 21 min 46 sec
CPU time	4 days 5 hours 13 min 21 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	2.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 09:49:35 (2123): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:25:52 (18457): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:27:43 (18524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:30:11 (18606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:31:18 (7410): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:20 (7410): No heartbeat from core client for 30 sec - exiting 20:31:21 (7410): No heartbeat from core client for 30 sec - exiting 20:31:22 (7410): No heartbeat from core client for 30 sec - exiting 20:31:23 (7410): No heartbeat from core client for 30 sec - exiting 20:31:24 (7410): No heartbeat from core client for 30 sec - exiting 20:31:25 (7410): No heartbeat from core client for 30 sec - exiting 20:31:26 (7410): No heartbeat from core client for 30 sec - exiting 20:31:27 (7410): No heartbeat from core client for 30 sec - exiting 20:31:28 (7410): No heartbeat from core client for 30 sec - exiting 20:31:29 (7410): No heartbeat from core client for 30 sec - exiting 20:31:30 (7410): No heartbeat from core client for 30 sec - exiting 20:31:31 (7410): No heartbeat from core client for 30 sec - exiting 20:31:32 (7410): No heartbeat from core client for 30 sec - exiting 20:31:33 (7410): No heartbeat from core client for 30 sec - exiting 20:31:34 (7410): No heartbeat from core client for 30 sec - exiting 20:31:35 (7410): No heartbeat from core client for 30 sec - exiting 20:31:36 (7410): No heartbeat from core client for 30 sec - exiting 20:31:37 (7410): No heartbeat from core client for 30 sec - exiting 20:31:38 (7410): No heartbeat from core client for 30 sec - exiting 20:31:39 (7410): No heartbeat from core client for 30 sec - exiting 20:31:40 (7410): No heartbeat from core client for 30 sec - exiting 20:31:41 (7410): No heartbeat from core client for 30 sec - exiting 20:31:42 (7410): No heartbeat from core client for 30 sec - exiting 20:31:43 (7410): No heartbeat from core client for 30 sec - exiting 20:31:44 (7410): No heartbeat from core client for 30 sec - exiting 20:31:45 (7410): No heartbeat from core client for 30 sec - exiting 20:31:46 (7410): No heartbeat from core client for 30 sec - exiting 23:16:21 (31147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:58 (31147): No heartbeat from core client for 30 sec - exiting 23:16:59 (31147): No heartbeat from core client for 30 sec - exiting 23:17:00 (31147): No heartbeat from core client for 30 sec - exiting 23:17:01 (31147): No heartbeat from core client for 30 sec - exiting 23:17:02 (31147): No heartbeat from core client for 30 sec - exiting 23:17:03 (31147): No heartbeat from core client for 30 sec - exiting 23:17:04 (31147): No heartbeat from core client for 30 sec - exiting 23:17:05 (31147): No heartbeat from core client for 30 sec - exiting 23:17:06 (31147): No heartbeat from core client for 30 sec - exiting 23:17:07 (31147): No heartbeat from core client for 30 sec - exiting 23:17:08 (31147): No heartbeat from core client for 30 sec - exiting 23:17:09 (31147): No heartbeat from core client for 30 sec - exiting 23:17:10 (31147): No heartbeat from core client for 30 sec - exiting 23:17:11 (31147): No heartbeat from core client for 30 sec - exiting 23:17:12 (31147): No heartbeat from core client for 30 sec - exiting 23:17:13 (31147): No heartbeat from core client for 30 sec - exiting 23:17:14 (31147): No heartbeat from core client for 30 sec - exiting 23:17:15 (31147): No heartbeat from core client for 30 sec - exiting 23:17:16 (31147): No heartbeat from core client for 30 sec - exiting 23:17:17 (31147): No heartbeat from core client for 30 sec - exiting 23:33:51 (6362): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:39:30 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:19 (2172): No heartbeat from core client for 30 sec - exiting 23:40:20 (2172): No heartbeat from core client for 30 sec - exiting 23:40:21 (2172): No heartbeat from core client for 30 sec - exiting 23:40:22 (2172): No heartbeat from core client for 30 sec - exiting 23:40:23 (2172): No heartbeat from core client for 30 sec - exiting 23:40:24 (2172): No heartbeat from core client for 30 sec - exiting 23:40:25 (2172): No heartbeat from core client for 30 sec - exiting 23:40:26 (2172): No heartbeat from core client for 30 sec - exiting 23:40:27 (2172): No heartbeat from core client for 30 sec - exiting 23:40:28 (2172): No heartbeat from core client for 30 sec - exiting 23:40:29 (2172): No heartbeat from core client for 30 sec - exiting 23:40:30 (2172): No heartbeat from core client for 30 sec - exiting 23:40:31 (2172): No heartbeat from core client for 30 sec - exiting 23:40:32 (2172): No heartbeat from core client for 30 sec - exiting 23:40:33 (2172): No heartbeat from core client for 30 sec - exiting 23:40:34 (2172): No heartbeat from core client for 30 sec - exiting 23:40:35 (2172): No heartbeat from core client for 30 sec - exiting 23:40:36 (2172): No heartbeat from core client for 30 sec - exiting 23:40:37 (2172): No heartbeat from core client for 30 sec - exiting 23:40:38 (2172): No heartbeat from core client for 30 sec - exiting 23:40:39 (2172): No heartbeat from core client for 30 sec - exiting 23:40:40 (2172): No heartbeat from core client for 30 sec - exiting 23:40:41 (2172): No heartbeat from core client for 30 sec - exiting 23:40:42 (2172): No heartbeat from core client for 30 sec - exiting 23:40:43 (2172): No heartbeat from core client for 30 sec - exiting 23:40:44 (2172): No heartbeat from core client for 30 sec - exiting 23:40:45 (2172): No heartbeat from core client for 30 sec - exiting 23:40:46 (2172): No heartbeat from core client for 30 sec - exiting 23:40:47 (2172): No heartbeat from core client for 30 sec - exiting 23:40:48 (2172): No heartbeat from core client for 30 sec - exiting 23:40:49 (2172): No heartbeat from core client for 30 sec - exiting 23:40:50 (2172): No heartbeat from core client for 30 sec - exiting 23:40:51 (2172): No heartbeat from core client for 30 sec - exiting 23:40:52 (2172): No heartbeat from core client for 30 sec - exiting 23:40:53 (2172): No heartbeat from core client for 30 sec - exiting 23:40:54 (2172): No heartbeat from core client for 30 sec - exiting 23:40:55 (2172): No heartbeat from core client for 30 sec - exiting 23:40:56 (2172): No heartbeat from core client for 30 sec - exiting 23:40:57 (2172): No heartbeat from core client for 30 sec - exiting 23:40:58 (2172): No heartbeat from core client for 30 sec - exiting 23:40:59 (2172): No heartbeat from core client for 30 sec - exiting 23:41:00 (2172): No heartbeat from core client for 30 sec - exiting 23:41:01 (2172): No heartbeat from core client for 30 sec - exiting 23:41:02 (2172): No heartbeat from core client for 30 sec - exiting 23:41:09 (2172): No heartbeat from core client for 30 sec - exiting 23:41:10 (2172): No heartbeat from core client for 30 sec - exiting 23:41:11 (2172): No heartbeat from core client for 30 sec - exiting 23:41:12 (2172): No heartbeat from core client for 30 sec - exiting 23:41:13 (2172): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:43:37 (2278): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:40 (2278): No heartbeat from core client for 30 sec - exiting 23:43:41 (2278): No heartbeat from core client for 30 sec - exiting 23:43:42 (2278): No heartbeat from core client for 30 sec - exiting 23:43:43 (2278): No heartbeat from core client for 30 sec - exiting 23:43:44 (2278): No heartbeat from core client for 30 sec - exiting 23:43:45 (2278): No heartbeat from core client for 30 sec - exiting 23:43:46 (2278): No heartbeat from core client for 30 sec - exiting 23:43:47 (2278): No heartbeat from core client for 30 sec - exiting 23:43:48 (2278): No heartbeat from core client for 30 sec - exiting 23:43:49 (2278): No heartbeat from core client for 30 sec - exiting 23:43:50 (2278): No heartbeat from core client for 30 sec - exiting 23:43:51 (2278): No heartbeat from core client for 30 sec - exiting 23:43:52 (2278): No heartbeat from core client for 30 sec - exiting 23:43:53 (2278): No heartbeat from core client for 30 sec - exiting 23:43:54 (2278): No heartbeat from core client for 30 sec - exiting 23:43:55 (2278): No heartbeat from core client for 30 sec - exiting 23:43:56 (2278): No heartbeat from core client for 30 sec - exiting 23:43:57 (2278): No heartbeat from core client for 30 sec - exiting 23:43:58 (2278): No heartbeat from core client for 30 sec - exiting 23:43:59 (2278): No heartbeat from core client for 30 sec - exiting 23:44:00 (2278): No heartbeat from core client for 30 sec - exiting 23:44:01 (2278): No heartbeat from core client for 30 sec - exiting 23:44:02 (2278): No heartbeat from core client for 30 sec - exiting 23:44:03 (2278): No heartbeat from core client for 30 sec - exiting 23:44:04 (2278): No heartbeat from core client for 30 sec - exiting 23:44:05 (2278): No heartbeat from core client for 30 sec - exiting 23:44:06 (2278): No heartbeat from core client for 30 sec - exiting 23:44:07 (2278): No heartbeat from core client for 30 sec - exiting 23:44:08 (2278): No heartbeat from core client for 30 sec - exiting 23:44:14 (2278): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 18:12:13 (1759): No heartbeat from core client for 30 sec - exiting 18:12:14 (1759): No heartbeat from core client for 30 sec - exiting 18:12:15 (1759): No heartbeat from core client for 30 sec - exiting 18:12:16 (1759): No heartbeat from core client for 30 sec - exiting 18:12:17 (1759): No heartbeat from core client for 30 sec - exiting 18:12:18 (1759): No heartbeat from core client for 30 sec - exiting 18:12:19 (1759): No heartbeat from core client for 30 sec - exiting 18:12:20 (1759): No heartbeat from core client for 30 sec - exiting 18:12:21 (1759): No heartbeat from core client for 30 sec - exiting 18:12:22 (1759): No heartbeat from core client for 30 sec - exiting 18:12:23 (1759): No heartbeat from core client for 30 sec - exiting 18:12:24 (1759): No heartbeat from core client for 30 sec - exiting 18:12:25 (1759): No heartbeat from core client for 30 sec - exiting 18:12:26 (1759): No heartbeat from core client for 30 sec - exiting 18:12:27 (1759): No heartbeat from core client for 30 sec - exiting 18:12:28 (1759): No heartbeat from core client for 30 sec - exiting 18:12:29 (1759): No heartbeat from core client for 30 sec - exiting 18:12:30 (1759): No heartbeat from core client for 30 sec - exiting 18:12:31 (1759): No heartbeat from core client for 30 sec - exiting 18:12:32 (1759): No heartbeat from core client for 30 sec - exiting 18:12:33 (1759): No heartbeat from core client for 30 sec - exiting 18:12:34 (1759): No heartbeat from core client for 30 sec - exiting 18:12:35 (1759): No heartbeat from core client for 30 sec - exiting 18:12:36 (1759): No heartbeat from core client for 30 sec - exiting 18:12:37 (1759): No heartbeat from core client for 30 sec - exiting 18:12:38 (1759): No heartbeat from core client for 30 sec - exiting 18:12:39 (1759): No heartbeat from core client for 30 sec - exiting 18:12:40 (1759): No heartbeat from core client for 30 sec - exiting 18:12:41 (1759): No heartbeat from core client for 30 sec - exiting 18:12:42 (1759): No heartbeat from core client for 30 sec - exiting 18:12:43 (1759): No heartbeat from core client for 30 sec - exiting 18:12:44 (1759): No heartbeat from core client for 30 sec - exiting 18:12:45 (1759): No heartbeat from core client for 30 sec - exiting 18:12:46 (1759): No heartbeat from core client for 30 sec - exiting 18:12:47 (1759): No heartbeat from core client for 30 sec - exiting 18:12:48 (1759): No heartbeat from core client for 30 sec - exiting 18:12:49 (1759): No heartbeat from core client for 30 sec - exiting 18:12:50 (1759): No heartbeat from core client for 30 sec - exiting 18:12:51 (1759): No heartbeat from core client for 30 sec - exiting 18:12:52 (1759): No heartbeat from core client for 30 sec - exiting 18:12:53 (1759): No heartbeat from core client for 30 sec - exiting 18:12:54 (1759): No heartbeat from core client for 30 sec - exiting 18:12:55 (1759): No heartbeat from core client for 30 sec - exiting 18:12:56 (1759): No heartbeat from core client for 30 sec - exiting 18:12:57 (1759): No heartbeat from core client for 30 sec - exiting 18:12:58 (1759): No heartbeat from core client for 30 sec - exiting 18:12:59 (1759): No heartbeat from core client for 30 sec - exiting 18:13:00 (1759): No heartbeat from core client for 30 sec - exiting 18:13:01 (1759): No heartbeat from core client for 30 sec - exiting 18:13:02 (1759): No heartbeat from core client for 30 sec - exiting 18:13:03 (1759): No heartbeat from core client for 30 sec - exiting 18:13:04 (1759): No heartbeat from core client for 30 sec - exiting 18:13:05 (1759): No heartbeat from core client for 30 sec - exiting 18:13:06 (1759): No heartbeat from core client for 30 sec - exiting 18:13:07 (1759): No heartbeat from core client for 30 sec - exiting 18:13:08 (1759): No heartbeat from core client for 30 sec - exiting 18:13:09 (1759): No heartbeat from core client for 30 sec - exiting 18:13:10 (1759): No heartbeat from core client for 30 sec - exiting 18:13:11 (1759): No heartbeat from core client for 30 sec - exiting 18:13:12 (1759): No heartbeat from core client for 30 sec - exiting 18:13:13 (1759): No heartbeat from core client for 30 sec - exiting 18:13:14 (1759): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:23:26 (28413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:40:24 (2537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:26 (2537): No heartbeat from core client for 30 sec - exiting 20:40:27 (2537): No heartbeat from core client for 30 sec - exiting 20:40:28 (2537): No heartbeat from core client for 30 sec - exiting 20:40:29 (2537): No heartbeat from core client for 30 sec - exiting 20:40:30 (2537): No heartbeat from core client for 30 sec - exiting 20:40:31 (2537): No heartbeat from core client for 30 sec - exiting 20:40:32 (2537): No heartbeat from core client for 30 sec - exiting 20:40:33 (2537): No heartbeat from core client for 30 sec - exiting 20:40:34 (2537): No heartbeat from core client for 30 sec - exiting 20:40:35 (2537): No heartbeat from core client for 30 sec - exiting 20:40:36 (2537): No heartbeat from core client for 30 sec - exiting 20:40:38 (2537): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:10:26 (1296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:28 (1296): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf773f400] [0xf773f430] /lib32/libc.so.6(gsignal+0x4f)[0xf757dc4f] /lib32/libc.so.6(abort+0x175)[0xf7581175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75690f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77de400] [0xf77de430] /lib32/libc.so.6(gsignal+0x4f)[0xf761cc4f] /lib32/libc.so.6(abort+0x175)[0xf7620175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf76080f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a4400] [0xf77a4430] /lib32/libc.so.6(gsignal+0x4f)[0xf75e2c4f] /lib32/libc.so.6(abort+0x175)[0xf75e6175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75ce0f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7768400] [0xf7768430] /lib32/libc.so.6(gsignal+0x4f)[0xf75a6c4f] /lib32/libc.so.6(abort+0x175)[0xf75aa175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75920f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf774b400] [0xf774b430] /lib32/libc.so.6(gsignal+0x4f)[0xf7589c4f] /lib32/libc.so.6(abort+0x175)[0xf758d175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75750f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77de400] [0xf77de430] /lib32/libc.so.6(gsignal+0x4f)[0xf761cc4f] /lib32/libc.so.6(abort+0x175)[0xf7620175] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf76080f3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Dec 2011 12:11:49	1181044	13661893	hadcm3n_ylrh_1900_40_007513844_3	129,600	328,442	2.5343
09 Dec 2011 08:32:35	1181044	13661893	hadcm3n_ylrh_1900_40_007513844_3	103,680	266,066	2.5662
07 Dec 2011 10:22:14	1181044	13661893	hadcm3n_ylrh_1900_40_007513844_3	77,760	202,125	2.5993
05 Dec 2011 20:15:06	1181044	13661893	hadcm3n_ylrh_1900_40_007513844_3	51,840	134,360	2.5918
03 Dec 2011 20:01:15	1181044	13661893	hadcm3n_ylrh_1900_40_007513844_3	25,920	67,341	2.5980