Task 13626659

Name	hadcm3n_o5az_1940_40_007543399_0
Workunit	7740631
Created	10 Nov 2011, 1:30:54 UTC
Sent	16 Nov 2011, 8:49:08 UTC
Report deadline	15 Feb 2012, 16:16:19 UTC
Received	29 Apr 2012, 16:46:31 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1124299
Run time	159 days 1 hours 56 min 32 sec
CPU time	121 days 10 hours 12 min 30 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	0.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:58:55 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:20:01 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:42:02 (1012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:33:12 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:33:14 (4044): No heartbeat from core client for 30 sec - exiting 12:44:51 (832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:18:43 (288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1356, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:29:42 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:43 (2224): No heartbeat from core client for 30 sec - exiting 19:29:44 (2224): No heartbeat from core client for 30 sec - exiting 19:29:45 (2224): No heartbeat from core client for 30 sec - exiting 19:29:46 (2224): No heartbeat from core client for 30 sec - exiting 19:29:47 (2224): No heartbeat from core client for 30 sec - exiting 19:29:48 (2224): No heartbeat from core client for 30 sec - exiting 19:29:49 (2224): No heartbeat from core client for 30 sec - exiting 19:29:51 (2224): No heartbeat from core client for 30 sec - exiting 19:29:52 (2224): No heartbeat from core client for 30 sec - exiting 19:29:53 (2224): No heartbeat from core client for 30 sec - exiting Ocean Restart file copy failed on o5azko.dah0c20 Suspended CPDN Monitor - Suspend request from BOINC... 12:29:18 (3896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:43 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:44 (3000): No heartbeat from core client for 30 sec - exiting 18:40:45 (3000): No heartbeat from core client for 30 sec - exiting 18:40:46 (3000): No heartbeat from core client for 30 sec - exiting 18:40:47 (3000): No heartbeat from core client for 30 sec - exiting 18:40:49 (3000): No heartbeat from core client for 30 sec - exiting 18:40:50 (3000): No heartbeat from core client for 30 sec - exiting 18:40:51 (3000): No heartbeat from core client for 30 sec - exiting 18:40:52 (3000): No heartbeat from core client for 30 sec - exiting 18:40:53 (3000): No heartbeat from core client for 30 sec - exiting 18:40:54 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:25:11 (4068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 23:50:43 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:50:45 (2144): No heartbeat from core client for 30 sec - exiting 23:50:46 (2144): No heartbeat from core client for 30 sec - exiting 23:50:47 (2144): No heartbeat from core client for 30 sec - exiting 23:50:48 (2144): No heartbeat from core client for 30 sec - exiting 23:50:49 (2144): No heartbeat from core client for 30 sec - exiting 23:50:50 (2144): No heartbeat from core client for 30 sec - exiting 23:50:51 (2144): No heartbeat from core client for 30 sec - exiting 23:50:52 (2144): No heartbeat from core client for 30 sec - exiting 23:50:53 (2144): No heartbeat from core client for 30 sec - exiting 23:50:54 (2144): No heartbeat from core client for 30 sec - exiting 02:33:24 (5508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:25 (5508): No heartbeat from core client for 30 sec - exiting 02:33:26 (5508): No heartbeat from core client for 30 sec - exiting 02:33:27 (5508): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... 02:17:34 (3860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... 22:43:16 (584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:40:03 (3432): No heartbeat from core client for 30 sec - exiting 10:40:04 (3432): No heartbeat from core client for 30 sec - exiting 10:40:05 (3432): No heartbeat from core client for 30 sec - exiting 10:40:06 (3432): No heartbeat from core client for 30 sec - exiting 10:40:07 (3432): No heartbeat from core client for 30 sec - exiting 10:40:08 (3432): No heartbeat from core client for 30 sec - exiting 10:40:09 (3432): No heartbeat from core client for 30 sec - exiting 10:40:10 (3432): No heartbeat from core client for 30 sec - exiting 10:40:11 (3432): No heartbeat from core client for 30 sec - exiting 10:40:13 (3432): No heartbeat from core client for 30 sec - exiting 10:40:14 (3432): No heartbeat from core client for 30 sec - exiting 10:40:15 (3432): No heartbeat from core client for 30 sec - exiting 10:40:16 (3432): No heartbeat from core client for 30 sec - exiting 10:40:17 (3432): No heartbeat from core client for 30 sec - exiting 10:40:18 (3432): No heartbeat from core client for 30 sec - exiting 10:40:19 (3432): No heartbeat from core client for 30 sec - exiting 10:40:20 (3432): No heartbeat from core client for 30 sec - exiting 10:40:21 (3432): No heartbeat from core client for 30 sec - exiting 10:40:22 (3432): No heartbeat from core client for 30 sec - exiting 10:40:23 (3432): No heartbeat from core client for 30 sec - exiting 10:40:25 (3432): No heartbeat from core client for 30 sec - exiting 10:40:26 (3432): No heartbeat from core client for 30 sec - exiting 10:40:27 (3432): No heartbeat from core client for 30 sec - exiting 10:40:28 (3432): No heartbeat from core client for 30 sec - exiting 10:40:29 (3432): No heartbeat from core client for 30 sec - exiting 10:40:30 (3432): No heartbeat from core client for 30 sec - exiting 10:40:31 (3432): No heartbeat from core client for 30 sec - exiting 10:40:32 (3432): No heartbeat from core client for 30 sec - exiting 10:40:33 (3432): No heartbeat from core client for 30 sec - exiting 10:40:34 (3432): No heartbeat from core client for 30 sec - exiting 10:40:36 (3432): No heartbeat from core client for 30 sec - exiting 10:40:37 (3432): No heartbeat from core client for 30 sec - exiting 10:40:38 (3432): No heartbeat from core client for 30 sec - exiting 10:40:39 (3432): No heartbeat from core client for 30 sec - exiting 10:40:40 (3432): No heartbeat from core client for 30 sec - exiting 10:40:41 (3432): No heartbeat from core client for 30 sec - exiting 10:40:42 (3432): No heartbeat from core client for 30 sec - exiting 10:40:43 (3432): No heartbeat from core client for 30 sec - exiting 10:40:44 (3432): No heartbeat from core client for 30 sec - exiting 10:40:45 (3432): No heartbeat from core client for 30 sec - exiting 10:40:46 (3432): No heartbeat from core client for 30 sec - exiting 10:40:48 (3432): No heartbeat from core client for 30 sec - exiting 10:40:49 (3432): No heartbeat from core client for 30 sec - exiting 10:40:50 (3432): No heartbeat from core client for 30 sec - exiting 10:40:51 (3432): No heartbeat from core client for 30 sec - exiting 10:40:52 (3432): No heartbeat from core client for 30 sec - exiting 10:40:53 (3432): No heartbeat from core client for 30 sec - exiting 10:40:54 (3432): No heartbeat from core client for 30 sec - exiting 10:40:55 (3432): No heartbeat from core client for 30 sec - exiting 10:40:56 (3432): No heartbeat from core client for 30 sec - exiting 10:40:57 (3432): No heartbeat from core client for 30 sec - exiting 10:40:58 (3432): No heartbeat from core client for 30 sec - exiting 10:41:00 (3432): No heartbeat from core client for 30 sec - exiting 10:41:01 (3432): No heartbeat from core client for 30 sec - exiting 10:41:02 (3432): No heartbeat from core client for 30 sec - exiting 10:41:03 (3432): No heartbeat from core client for 30 sec - exiting 10:41:04 (3432): No heartbeat from core client for 30 sec - exiting 10:41:05 (3432): No heartbeat from core client for 30 sec - exiting 10:41:06 (3432): No heartbeat from core client for 30 sec - exiting 10:41:07 (3432): No heartbeat from core client for 30 sec - exiting 10:41:08 (3432): No heartbeat from core client for 30 sec - exiting 10:41:09 (3432): No heartbeat from core client for 30 sec - exiting 10:41:11 (3432): No heartbeat from core client for 30 sec - exiting 10:41:12 (3432): No heartbeat from core client for 30 sec - exiting 10:41:13 (3432): No heartbeat from core client for 30 sec - exiting 10:41:14 (3432): No heartbeat from core client for 30 sec - exiting 10:41:15 (3432): No heartbeat from core client for 30 sec - exiting 10:41:16 (3432): No heartbeat from core client for 30 sec - exiting 10:41:17 (3432): No heartbeat from core client for 30 sec - exiting 10:41:18 (3432): No heartbeat from core client for 30 sec - exiting 10:41:19 (3432): No heartbeat from core client for 30 sec - exiting 10:41:20 (3432): No heartbeat from core client for 30 sec - exiting 10:41:21 (3432): No heartbeat from core client for 30 sec - exiting 10:41:23 (3432): No heartbeat from core client for 30 sec - exiting 10:41:24 (3432): No heartbeat from core client for 30 sec - exiting 10:41:25 (3432): No heartbeat from core client for 30 sec - exiting 10:41:26 (3432): No heartbeat from core client for 30 sec - exiting 10:41:27 (3432): No heartbeat from core client for 30 sec - exiting 10:41:28 (3432): No heartbeat from core client for 30 sec - exiting 10:41:29 (3432): No heartbeat from core client for 30 sec - exiting 10:41:30 (3432): No heartbeat from core client for 30 sec - exiting 10:41:31 (3432): No heartbeat from core client for 30 sec - exiting 10:41:32 (3432): No heartbeat from core client for 30 sec - exiting 10:41:33 (3432): No heartbeat from core client for 30 sec - exiting 10:41:35 (3432): No heartbeat from core client for 30 sec - exiting 10:41:36 (3432): No heartbeat from core client for 30 sec - exiting 10:41:37 (3432): No heartbeat from core client for 30 sec - exiting 10:41:38 (3432): No heartbeat from core client for 30 sec - exiting 10:41:39 (3432): No heartbeat from core client for 30 sec - exiting 10:41:40 (3432): No heartbeat from core client for 30 sec - exiting 10:41:41 (3432): No heartbeat from core client for 30 sec - exiting 10:41:42 (3432): No heartbeat from core client for 30 sec - exiting 10:41:43 (3432): No heartbeat from core client for 30 sec - exiting 10:41:44 (3432): No heartbeat from core client for 30 sec - exiting 10:41:46 (3432): No heartbeat from core client for 30 sec - exiting 10:41:47 (3432): No heartbeat from core client for 30 sec - exiting 10:41:48 (3432): No heartbeat from core client for 30 sec - exiting 10:41:49 (3432): No heartbeat from core client for 30 sec - exiting 10:41:50 (3432): No heartbeat from core client for 30 sec - exiting 10:41:51 (3432): No heartbeat from core client for 30 sec - exiting 10:41:52 (3432): No heartbeat from core client for 30 sec - exiting 10:41:53 (3432): No heartbeat from core client for 30 sec - exiting 10:41:54 (3432): No heartbeat from core client for 30 sec - exiting 10:41:55 (3432): No heartbeat from core client for 30 sec - exiting 10:41:56 (3432): No heartbeat from core client for 30 sec - exiting 10:41:58 (3432): No heartbeat from core client for 30 sec - exiting 10:41:59 (3432): No heartbeat from core client for 30 sec - exiting 10:42:00 (3432): No heartbeat from core client for 30 sec - exiting 10:42:01 (3432): No heartbeat from core client for 30 sec - exiting 10:42:02 (3432): No heartbeat from core client for 30 sec - exiting 10:42:03 (3432): No heartbeat from core client for 30 sec - exiting 10:42:04 (3432): No heartbeat from core client for 30 sec - exiting 10:42:05 (3432): No heartbeat from core client for 30 sec - exiting 10:42:06 (3432): No heartbeat from core client for 30 sec - exiting 10:42:07 (3432): No heartbeat from core client for 30 sec - exiting 10:42:08 (3432): No heartbeat from core client for 30 sec - exiting 10:42:10 (3432): No heartbeat from core client for 30 sec - exiting 10:42:11 (3432): No heartbeat from core client for 30 sec - exiting 10:42:12 (3432): No heartbeat from core client for 30 sec - exiting 10:42:13 (3432): No heartbeat from core client for 30 sec - exiting 10:42:14 (3432): No heartbeat from core client for 30 sec - exiting 10:42:15 (3432): No heartbeat from core client for 30 sec - exiting 10:42:16 (3432): No heartbeat from core client for 30 sec - exiting 10:42:17 (3432): No heartbeat from core client for 30 sec - exiting 10:42:18 (3432): No heartbeat from core client for 30 sec - exiting 10:42:19 (3432): No heartbeat from core client for 30 sec - exiting 10:42:21 (3432): No heartbeat from core client for 30 sec - exiting 10:42:22 (3432): No heartbeat from core client for 30 sec - exiting 10:42:23 (3432): No heartbeat from core client for 30 sec - exiting 10:42:24 (3432): No heartbeat from core client for 30 sec - exiting 10:42:25 (3432): No heartbeat from core client for 30 sec - exiting 10:42:26 (3432): No heartbeat from core client for 30 sec - exiting 10:42:27 (3432): No heartbeat from core client for 30 sec - exiting 10:42:28 (3432): No heartbeat from core client for 30 sec - exiting 10:42:29 (3432): No heartbeat from core client for 30 sec - exiting 10:42:30 (3432): No heartbeat from core client for 30 sec - exiting 10:42:31 (3432): No heartbeat from core client for 30 sec - exiting 10:42:33 (3432): No heartbeat from core client for 30 sec - exiting 10:42:34 (3432): No heartbeat from core client for 30 sec - exiting 10:42:35 (3432): No heartbeat from core client for 30 sec - exiting 10:42:36 (3432): No heartbeat from core client for 30 sec - exiting 10:42:37 (3432): No heartbeat from core client for 30 sec - exiting 10:42:38 (3432): No heartbeat from core client for 30 sec - exiting 10:42:39 (3432): No heartbeat from core client for 30 sec - exiting 10:42:40 (3432): No heartbeat from core client for 30 sec - exiting 10:42:41 (3432): No heartbeat from core client for 30 sec - exiting 10:42:42 (3432): No heartbeat from core client for 30 sec - exiting 10:42:43 (3432): No heartbeat from core client for 30 sec - exiting 10:42:45 (3432): No heartbeat from core client for 30 sec - exiting 10:42:46 (3432): No heartbeat from core client for 30 sec - exiting 10:42:47 (3432): No heartbeat from core client for 30 sec - exiting 10:42:48 (3432): No heartbeat from core client for 30 sec - exiting 10:42:49 (3432): No heartbeat from core client for 30 sec - exiting 10:42:50 (3432): No heartbeat from core client for 30 sec - exiting 10:42:51 (3432): No heartbeat from core client for 30 sec - exiting 10:42:52 (3432): No heartbeat from core client for 30 sec - exiting 10:42:53 (3432): No heartbeat from core client for 30 sec - exiting 10:42:54 (3432): No heartbeat from core client for 30 sec - exiting 10:42:56 (3432): No heartbeat from core client for 30 sec - exiting 10:42:57 (3432): No heartbeat from core client for 30 sec - exiting 10:42:58 (3432): No heartbeat from core client for 30 sec - exiting 10:42:59 (3432): No heartbeat from core client for 30 sec - exiting 10:43:00 (3432): No heartbeat from core client for 30 sec - exiting 10:43:01 (3432): No heartbeat from core client for 30 sec - exiting 10:43:02 (3432): No heartbeat from core client for 30 sec - exiting 10:43:03 (3432): No heartbeat from core client for 30 sec - exiting 10:43:04 (3432): No heartbeat from core client for 30 sec - exiting 10:43:05 (3432): No heartbeat from core client for 30 sec - exiting 10:43:06 (3432): No heartbeat from core client for 30 sec - exiting 10:43:08 (3432): No heartbeat from core client for 30 sec - exiting 10:43:09 (3432): No heartbeat from core client for 30 sec - exiting 10:43:10 (3432): No heartbeat from core client for 30 sec - exiting 10:43:11 (3432): No heartbeat from core client for 30 sec - exiting 10:43:12 (3432): No heartbeat from core client for 30 sec - exiting 10:43:13 (3432): No heartbeat from core client for 30 sec - exiting 10:43:14 (3432): No heartbeat from core client for 30 sec - exiting 10:43:15 (3432): No heartbeat from core client for 30 sec - exiting 10:43:16 (3432): No heartbeat from core client for 30 sec - exiting 10:43:17 (3432): No heartbeat from core client for 30 sec - exiting 10:43:18 (3432): No heartbeat from core client for 30 sec - exiting 10:43:20 (3432): No heartbeat from core client for 30 sec - exiting 10:43:21 (3432): No heartbeat from core client for 30 sec - exiting 10:43:22 (3432): No heartbeat from core client for 30 sec - exiting 10:43:23 (3432): No heartbeat from core client for 30 sec - exiting 10:43:24 (3432): No heartbeat from core client for 30 sec - exiting 10:43:25 (3432): No heartbeat from core client for 30 sec - exiting 10:43:26 (3432): No heartbeat from core client for 30 sec - exiting 10:43:27 (3432): No heartbeat from core client for 30 sec - exiting 10:43:28 (3432): No heartbeat from core client for 30 sec - exiting 10:43:29 (3432): No heartbeat from core client for 30 sec - exiting 10:43:31 (3432): No heartbeat from core client for 30 sec - exiting 10:43:32 (3432): No heartbeat from core client for 30 sec - exiting 10:43:33 (3432): No heartbeat from core client for 30 sec - exiting 10:43:34 (3432): No heartbeat from core client for 30 sec - exiting 10:43:35 (3432): No heartbeat from core client for 30 sec - exiting 10:43:36 (3432): No heartbeat from core client for 30 sec - exiting 10:43:37 (3432): No heartbeat from core client for 30 sec - exiting 10:43:38 (3432): No heartbeat from core client for 30 sec - exiting 10:43:39 (3432): No heartbeat from core client for 30 sec - exiting 10:43:40 (3432): No heartbeat from core client for 30 sec - exiting 10:43:41 (3432): No heartbeat from core client for 30 sec - exiting 10:43:43 (3432): No heartbeat from core client for 30 sec - exiting 10:43:44 (3432): No heartbeat from core client for 30 sec - exiting 10:43:45 (3432): No heartbeat from core client for 30 sec - exiting 10:43:46 (3432): No heartbeat from core client for 30 sec - exiting 10:43:47 (3432): No heartbeat from core client for 30 sec - exiting 10:43:48 (3432): No heartbeat from core client for 30 sec - exiting 10:43:49 (3432): No heartbeat from core client for 30 sec - exiting 10:43:50 (3432): No heartbeat from core client for 30 sec - exiting 10:43:51 (3432): No heartbeat from core client for 30 sec - exiting 10:43:52 (3432): No heartbeat from core client for 30 sec - exiting 10:43:53 (3432): No heartbeat from core client for 30 sec - exiting 10:43:55 (3432): No heartbeat from core client for 30 sec - exiting 10:43:56 (3432): No heartbeat from core client for 30 sec - exiting 10:43:57 (3432): No heartbeat from core client for 30 sec - exiting 10:43:58 (3432): No heartbeat from core client for 30 sec - exiting 10:43:59 (3432): No heartbeat from core client for 30 sec - exiting 10:44:00 (3432): No heartbeat from core client for 30 sec - exiting 10:44:01 (3432): No heartbeat from core client for 30 sec - exiting 10:44:02 (3432): No heartbeat from core client for 30 sec - exiting 10:44:03 (3432): No heartbeat from core client for 30 sec - exiting 10:44:04 (3432): No heartbeat from core client for 30 sec - exiting 10:44:06 (3432): No heartbeat from core client for 30 sec - exiting 10:44:07 (3432): No heartbeat from core client for 30 sec - exiting 10:44:08 (3432): No heartbeat from core client for 30 sec - exiting 10:44:09 (3432): No heartbeat from core client for 30 sec - exiting 10:44:10 (3432): No heartbeat from core client for 30 sec - exiting 10:44:11 (3432): No heartbeat from core client for 30 sec - exiting 10:44:12 (3432): No heartbeat from core client for 30 sec - exiting 10:44:13 (3432): No heartbeat from core client for 30 sec - exiting 10:44:14 (3432): No heartbeat from core client for 30 sec - exiting 10:44:15 (3432): No heartbeat from core client for 30 sec - exiting 10:44:16 (3432): No heartbeat from core client for 30 sec - exiting 10:44:18 (3432): No heartbeat from core client for 30 sec - exiting 10:44:19 (3432): No heartbeat from core client for 30 sec - exiting 10:44:20 (3432): No heartbeat from core client for 30 sec - exiting 10:44:21 (3432): No heartbeat from core client for 30 sec - exiting 10:44:22 (3432): No heartbeat from core client for 30 sec - exiting 10:44:23 (3432): No heartbeat from core client for 30 sec - exiting 10:44:24 (3432): No heartbeat from core client for 30 sec - exiting 10:44:25 (3432): No heartbeat from core client for 30 sec - exiting 10:44:26 (3432): No heartbeat from core client for 30 sec - exiting 10:44:27 (3432): No heartbeat from core client for 30 sec - exiting 10:44:28 (3432): No heartbeat from core client for 30 sec - exiting 10:44:30 (3432): No heartbeat from core client for 30 sec - exiting 10:44:31 (3432): No heartbeat from core client for 30 sec - exiting 10:44:32 (3432): No heartbeat from core client for 30 sec - exiting 10:44:33 (3432): No heartbeat from core client for 30 sec - exiting 10:44:34 (3432): No heartbeat from core client for 30 sec - exiting 10:44:35 (3432): No heartbeat from core client for 30 sec - exiting 10:44:36 (3432): No heartbeat from core client for 30 sec - exiting 10:44:37 (3432): No heartbeat from core client for 30 sec - exiting 10:44:38 (3432): No heartbeat from core client for 30 sec - exiting 10:44:39 (3432): No heartbeat from core client for 30 sec - exiting 10:44:41 (3432): No heartbeat from core client for 30 sec - exiting 10:44:42 (3432): No heartbeat from core client for 30 sec - exiting 10:44:43 (3432): No heartbeat from core client for 30 sec - exiting 10:44:44 (3432): No heartbeat from core client for 30 sec - exiting 10:44:45 (3432): No heartbeat from core client for 30 sec - exiting 10:44:46 (3432): No heartbeat from core client for 30 sec - exiting 10:44:47 (3432): No heartbeat from core client for 30 sec - exiting 10:44:48 (3432): No heartbeat from core client for 30 sec - exiting 10:44:49 (3432): No heartbeat from core client for 30 sec - exiting 10:44:50 (3432): No heartbeat from core client for 30 sec - exiting 10:44:51 (3432): No heartbeat from core client for 30 sec - exiting 10:44:53 (3432): No heartbeat from core client for 30 sec - exiting 10:44:54 (3432): No heartbeat from core client for 30 sec - exiting 10:44:55 (3432): No heartbeat from core client for 30 sec - exiting 10:44:56 (3432): No heartbeat from core client for 30 sec - exiting 10:44:57 (3432): No heartbeat from core client for 30 sec - exiting 10:44:58 (3432): No heartbeat from core client for 30 sec - exiting 10:44:59 (3432): No heartbeat from core client for 30 sec - exiting 10:45:00 (3432): No heartbeat from core client for 30 sec - exiting 10:45:01 (3432): No heartbeat from core client for 30 sec - exiting 10:45:02 (3432): No heartbeat from core client for 30 sec - exiting 10:45:03 (3432): No heartbeat from core client for 30 sec - exiting 10:45:05 (3432): No heartbeat from core client for 30 sec - exiting 10:45:06 (3432): No heartbeat from core client for 30 sec - exiting 10:45:07 (3432): No heartbeat from core client for 30 sec - exiting 10:45:08 (3432): No heartbeat from core client for 30 sec - exiting 10:45:09 (3432): No heartbeat from core client for 30 sec - exiting 10:45:10 (3432): No heartbeat from core client for 30 sec - exiting 10:45:11 (3432): No heartbeat from core client for 30 sec - exiting 10:45:12 (3432): No heartbeat from core client for 30 sec - exiting 10:45:13 (3432): No heartbeat from core client for 30 sec - exiting 10:45:14 (3432): No heartbeat from core client for 30 sec - exiting 10:45:16 (3432): No heartbeat from core client for 30 sec - exiting 10:45:17 (3432): No heartbeat from core client for 30 sec - exiting 10:45:18 (3432): No heartbeat from core client for 30 sec - exiting 10:45:19 (3432): No heartbeat from core client for 30 sec - exiting 10:45:20 (3432): No heartbeat from core client for 30 sec - exiting 10:45:21 (3432): No heartbeat from core client for 30 sec - exiting 10:45:22 (3432): No heartbeat from core client for 30 sec - exiting 10:45:23 (3432): No heartbeat from core client for 30 sec - exiting 10:45:24 (3432): No heartbeat from core client for 30 sec - exiting 10:45:25 (3432): No heartbeat from core client for 30 sec - exiting 10:45:26 (3432): No heartbeat from core client for 30 sec - exiting 10:45:28 (3432): No heartbeat from core client for 30 sec - exiting 10:45:29 (3432): No heartbeat from core client for 30 sec - exiting 10:45:30 (3432): No heartbeat from core client for 30 sec - exiting 10:45:31 (3432): No heartbeat from core client for 30 sec - exiting 10:45:32 (3432): No heartbeat from core client for 30 sec - exiting 10:45:33 (3432): No heartbeat from core client for 30 sec - exiting 10:45:34 (3432): No heartbeat from core client for 30 sec - exiting 10:45:35 (3432): No heartbeat from core client for 30 sec - exiting 10:45:36 (3432): No heartbeat from core client for 30 sec - exiting 10:45:37 (3432): No heartbeat from core client for 30 sec - exiting 10:45:38 (3432): No heartbeat from core client for 30 sec - exiting 10:45:40 (3432): No heartbeat from core client for 30 sec - exiting 10:45:41 (3432): No heartbeat from core client for 30 sec - exiting 10:45:42 (3432): No heartbeat from core client for 30 sec - exiting 10:45:43 (3432): No heartbeat from core client for 30 sec - exiting 10:45:44 (3432): No heartbeat from core client for 30 sec - exiting 10:45:45 (3432): No heartbeat from core client for 30 sec - exiting 10:45:46 (3432): No heartbeat from core client for 30 sec - exiting 10:45:47 (3432): No heartbeat from core client for 30 sec - exiting 10:45:48 (3432): No heartbeat from core client for 30 sec - exiting 10:45:49 (3432): No heartbeat from core client for 30 sec - exiting 10:45:51 (3432): No heartbeat from core client for 30 sec - exiting 10:45:52 (3432): No heartbeat from core client for 30 sec - exiting 10:45:53 (3432): No heartbeat from core client for 30 sec - exiting 10:45:54 (3432): No heartbeat from core client for 30 sec - exiting 10:45:55 (3432): No heartbeat from core client for 30 sec - exiting 10:45:56 (3432): No heartbeat from core client for 30 sec - exiting 10:45:57 (3432): No heartbeat from core client for 30 sec - exiting 10:45:58 (3432): No heartbeat from core client for 30 sec - exiting 10:45:59 (3432): No heartbeat from core client for 30 sec - exiting 10:46:00 (3432): No heartbeat from core client for 30 sec - exiting 10:46:01 (3432): No heartbeat from core client for 30 sec - exiting 10:46:03 (3432): No heartbeat from core client for 30 sec - exiting 10:46:04 (3432): No heartbeat from core client for 30 sec - exiting 10:46:05 (3432): No heartbeat from core client for 30 sec - exiting 10:46:06 (3432): No heartbeat from core client for 30 sec - exiting 10:46:07 (3432): No heartbeat from core client for 30 sec - exiting 10:46:08 (3432): No heartbeat from core client for 30 sec - exiting 10:46:09 (3432): No heartbeat from core client for 30 sec - exiting 10:46:10 (3432): No heartbeat from core client for 30 sec - exiting 10:46:11 (3432): No heartbeat from core client for 30 sec - exiting 10:46:12 (3432): No heartbeat from core client for 30 sec - exiting 10:46:13 (3432): No heartbeat from core client for 30 sec - exiting 10:46:15 (3432): No heartbeat from core client for 30 sec - exiting 10:46:16 (3432): No heartbeat from core client for 30 sec - exiting 10:46:17 (3432): No heartbeat from core client for 30 sec - exiting 10:46:18 (3432): No heartbeat from core client for 30 sec - exiting 10:46:19 (3432): No heartbeat from core client for 30 sec - exiting 10:46:20 (3432): No heartbeat from core client for 30 sec - exiting 10:46:21 (3432): No heartbeat from core client for 30 sec - exiting 10:46:22 (3432): No heartbeat from core client for 30 sec - exiting 10:46:23 (3432): No heartbeat from core client for 30 sec - exiting 10:46:24 (3432): No heartbeat from core client for 30 sec - exiting 10:46:26 (3432): No heartbeat from core client for 30 sec - exiting 10:46:27 (3432): No heartbeat from core client for 30 sec - exiting 10:46:28 (3432): No heartbeat from core client for 30 sec - exiting 10:46:29 (3432): No heartbeat from core client for 30 sec - exiting 10:46:30 (3432): No heartbeat from core client for 30 sec - exiting 10:46:31 (3432): No heartbeat from core client for 30 sec - exiting 10:46:32 (3432): No heartbeat from core client for 30 sec - exiting 10:46:33 (3432): No heartbeat from core client for 30 sec - exiting 10:46:34 (3432): No heartbeat from core client for 30 sec - exiting 10:46:35 (3432): No heartbeat from core client for 30 sec - exiting 10:46:36 (3432): No heartbeat from core client for 30 sec - exiting 10:46:38 (3432): No heartbeat from core client for 30 sec - exiting 10:46:39 (3432): No heartbeat from core client for 30 sec - exiting 10:46:40 (3432): No heartbeat from core client for 30 sec - exiting 10:46:41 (3432): No heartbeat from core client for 30 sec - exiting 10:46:42 (3432): No heartbeat from core client for 30 sec - exiting 10:46:43 (3432): No heartbeat from core client for 30 sec - exiting 10:46:44 (3432): No heartbeat from core client for 30 sec - exiting 10:46:45 (3432): No heartbeat from core client for 30 sec - exiting 10:46:46 (3432): No heartbeat from core client for 30 sec - exiting 10:46:47 (3432): No heartbeat from core client for 30 sec - exiting 10:46:48 (3432): No heartbeat from core client for 30 sec - exiting 10:46:50 (3432): No heartbeat from core client for 30 sec - exiting 10:46:51 (3432): No heartbeat from core client for 30 sec - exiting 10:46:52 (3432): No heartbeat from core client for 30 sec - exiting 10:46:53 (3432): No heartbeat from core client for 30 sec - exiting 10:46:54 (3432): No heartbeat from core client for 30 sec - exiting 10:46:55 (3432): No heartbeat from core client for 30 sec - exiting 10:46:56 (3432): No heartbeat from core client for 30 sec - exiting 10:46:57 (3432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:05:34 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:04 (1028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:50:31 (4532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:42 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 16:50:10 (2824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:00 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Apr 2012 15:00:10	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	1,036,800	10,487,006	10.1148
24 Apr 2012 02:33:37	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	1,010,880	10,179,132	10.0696
20 Apr 2012 02:35:22	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	984,960	9,880,050	10.0309
14 Apr 2012 20:08:29	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	959,040	9,600,917	10.0110
10 Apr 2012 11:31:30	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	933,120	9,329,830	9.9985
06 Apr 2012 08:30:51	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	907,200	9,088,055	10.0177
02 Apr 2012 07:57:28	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	881,280	8,856,515	10.0496
29 Mar 2012 20:42:07	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	855,360	8,854,661	10.3520
25 Mar 2012 16:15:25	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	829,440	8,582,806	10.3477
21 Mar 2012 18:43:49	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	803,520	8,309,384	10.3412
17 Mar 2012 20:12:13	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	777,600	8,031,404	10.3285
14 Mar 2012 02:07:23	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	751,680	7,749,952	10.3102
09 Mar 2012 23:00:40	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	725,760	7,465,108	10.2859
06 Mar 2012 03:16:25	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	699,840	7,190,593	10.2746
02 Mar 2012 12:07:39	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	673,920	6,920,868	10.2696
27 Feb 2012 14:13:36	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	648,000	6,960,522	10.7415
24 Feb 2012 05:04:31	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	622,080	6,681,277	10.7402
19 Feb 2012 19:43:07	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	596,160	6,570,077	11.0207
16 Feb 2012 07:53:05	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	570,240	6,289,186	11.0290
13 Feb 2012 05:32:21	1124299	13626659	hadcm3n_o5az_1940_40_007543399_0	544,320	6,035,735	11.0886