Task 12738690

Name	hadcm3n_o2n0_1900_40_007198751_0
Workunit	7397031
Created	28 Mar 2011, 14:04:13 UTC
Sent	31 Mar 2011, 22:24:12 UTC
Report deadline	1 Jul 2011, 5:51:23 UTC
Received	7 Jun 2011, 9:21:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1106381
Run time	32 days 10 hours 1 min 58 sec
CPU time	29 days 18 hours 59 min 7 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:29:27 (5912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5556, selfPID=5556, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5528, selfPID=5528, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5596, selfPID=5596, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3840, selfPID=3840, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4596, selfPID=4596, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2612, selfPID=2612, iMonCtr=1 15:35:58 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:59 (5900): No heartbeat from core client for 30 sec - exiting 15:36:00 (5900): No heartbeat from core client for 30 sec - exiting 15:36:01 (5900): No heartbeat from core client for 30 sec - exiting 15:36:02 (5900): No heartbeat from core client for 30 sec - exiting 15:36:03 (5900): No heartbeat from core client for 30 sec - exiting 15:36:04 (5900): No heartbeat from core client for 30 sec - exiting 15:36:06 (5900): No heartbeat from core client for 30 sec - exiting 15:36:07 (5900): No heartbeat from core client for 30 sec - exiting 15:36:08 (5900): No heartbeat from core client for 30 sec - exiting 15:36:09 (5900): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5776, selfPID=5776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3440, selfPID=3440, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4152, selfPID=4152, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5736, selfPID=5736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5704, selfPID=5704, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1640, selfPID=1640, iMonCtr=1 06:34:58 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:59 (5516): No heartbeat from core client for 30 sec - exiting 06:35:00 (5516): No heartbeat from core client for 30 sec - exiting 06:35:01 (5516): No heartbeat from core client for 30 sec - exiting 06:35:02 (5516): No heartbeat from core client for 30 sec - exiting 06:35:03 (5516): No heartbeat from core client for 30 sec - exiting 06:35:04 (5516): No heartbeat from core client for 30 sec - exiting 06:35:05 (5516): No heartbeat from core client for 30 sec - exiting 06:35:06 (5516): No heartbeat from core client for 30 sec - exiting 06:35:07 (5516): No heartbeat from core client for 30 sec - exiting 06:35:08 (5516): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5088, selfPID=5088, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2000, selfPID=2000, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4488, selfPID=4488, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4912, selfPID=4912, iMonCtr=1 11:36:49 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:50 (4084): No heartbeat from core client for 30 sec - exiting 11:36:51 (4084): No heartbeat from core client for 30 sec - exiting 11:36:52 (4084): No heartbeat from core client for 30 sec - exiting 11:36:53 (4084): No heartbeat from core client for 30 sec - exiting 11:36:54 (4084): No heartbeat from core client for 30 sec - exiting 11:36:55 (4084): No heartbeat from core client for 30 sec - exiting 11:36:56 (4084): No heartbeat from core client for 30 sec - exiting 11:36:57 (4084): No heartbeat from core client for 30 sec - exiting 11:36:58 (4084): No heartbeat from core client for 30 sec - exiting 11:36:59 (4084): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5056, selfPID=5056, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4804, selfPID=4804, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3760, selfPID=3760, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3976, selfPID=3976, iMonCtr=1 01:35:46 (1956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:47 (1956): No heartbeat from core client for 30 sec - exiting 01:35:48 (1956): No heartbeat from core client for 30 sec - exiting 01:35:49 (1956): No heartbeat from core client for 30 sec - exiting 01:35:50 (1956): No heartbeat from core client for 30 sec - exiting 01:35:51 (1956): No heartbeat from core client for 30 sec - exiting 01:35:52 (1956): No heartbeat from core client for 30 sec - exiting 01:35:53 (1956): No heartbeat from core client for 30 sec - exiting 01:35:54 (1956): No heartbeat from core client for 30 sec - exiting 01:35:55 (1956): No heartbeat from core client for 30 sec - exiting 01:35:56 (1956): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4616, selfPID=4616, iMonCtr=1 09:34:46 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:47 (4700): No heartbeat from core client for 30 sec - exiting 09:34:48 (4700): No heartbeat from core client for 30 sec - exiting 09:34:49 (4700): No heartbeat from core client for 30 sec - exiting 09:34:50 (4700): No heartbeat from core client for 30 sec - exiting 09:34:51 (4700): No heartbeat from core client for 30 sec - exiting 09:34:52 (4700): No heartbeat from core client for 30 sec - exiting 09:34:53 (4700): No heartbeat from core client for 30 sec - exiting 09:34:54 (4700): No heartbeat from core client for 30 sec - exiting 09:34:55 (4700): No heartbeat from core client for 30 sec - exiting 09:34:56 (4700): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3020, selfPID=3020, iMonCtr=1 16:33:44 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:33:45 (3588): No heartbeat from core client for 30 sec - exiting 16:33:46 (3588): No heartbeat from core client for 30 sec - exiting 16:33:47 (3588): No heartbeat from core client for 30 sec - exiting 16:33:48 (3588): No heartbeat from core client for 30 sec - exiting 16:33:49 (3588): No heartbeat from core client for 30 sec - exiting 16:33:50 (3588): No heartbeat from core client for 30 sec - exiting 16:33:51 (3588): No heartbeat from core client for 30 sec - exiting 16:33:52 (3588): No heartbeat from core client for 30 sec - exiting 16:33:53 (3588): No heartbeat from core client for 30 sec - exiting 16:33:54 (3588): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3628, selfPID=3628, iMonCtr=1 01:00:10 (3652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:11 (3652): No heartbeat from core client for 30 sec - exiting 01:00:12 (3652): No heartbeat from core client for 30 sec - exiting 01:00:13 (3652): No heartbeat from core client for 30 sec - exiting 01:00:14 (3652): No heartbeat from core client for 30 sec - exiting 01:00:15 (3652): No heartbeat from core client for 30 sec - exiting 01:00:16 (3652): No heartbeat from core client for 30 sec - exiting 01:00:17 (3652): No heartbeat from core client for 30 sec - exiting 01:00:18 (3652): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. 06:16:38 (2160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:39 (2160): No heartbeat from core client for 30 sec - exiting 06:16:40 (2160): No heartbeat from core client for 30 sec - exiting 06:16:41 (2160): No heartbeat from core client for 30 sec - exiting 06:16:42 (2160): No heartbeat from core client for 30 sec - exiting 06:16:43 (2160): No heartbeat from core client for 30 sec - exiting 06:16:44 (2160): No heartbeat from core client for 30 sec - exiting 06:16:45 (2160): No heartbeat from core client for 30 sec - exiting 06:16:46 (2160): No heartbeat from core client for 30 sec - exiting 06:16:47 (2160): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3300, selfPID=3300, iMonCtr=1 18:53:39 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:40 (4568): No heartbeat from core client for 30 sec - exiting 18:53:41 (4568): No heartbeat from core client for 30 sec - exiting 18:53:42 (4568): No heartbeat from core client for 30 sec - exiting 18:53:43 (4568): No heartbeat from core client for 30 sec - exiting 18:53:44 (4568): No heartbeat from core client for 30 sec - exiting 18:53:45 (4568): No heartbeat from core client for 30 sec - exiting 18:53:46 (4568): No heartbeat from core client for 30 sec - exiting 18:53:47 (4568): No heartbeat from core client for 30 sec - exiting 18:53:48 (4568): No heartbeat from core client for 30 sec - exiting 18:53:49 (4568): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4844, selfPID=4844, iMonCtr=1 19:52:37 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:38 (664): No heartbeat from core client for 30 sec - exiting 19:52:39 (664): No heartbeat from core client for 30 sec - exiting 19:52:40 (664): No heartbeat from core client for 30 sec - exiting 19:52:41 (664): No heartbeat from core client for 30 sec - exiting 19:52:42 (664): No heartbeat from core client for 30 sec - exiting 19:52:43 (664): No heartbeat from core client for 30 sec - exiting 19:52:44 (664): No heartbeat from core client for 30 sec - exiting 19:52:45 (664): No heartbeat from core client for 30 sec - exiting 19:52:46 (664): No heartbeat from core client for 30 sec - exiting 19:52:47 (664): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5008, selfPID=5008, iMonCtr=1 17:51:36 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:51:37 (4716): No heartbeat from core client for 30 sec - exiting 17:51:38 (4716): No heartbeat from core client for 30 sec - exiting 17:51:39 (4716): No heartbeat from core client for 30 sec - exiting 17:51:40 (4716): No heartbeat from core client for 30 sec - exiting 17:51:41 (4716): No heartbeat from core client for 30 sec - exiting 17:51:42 (4716): No heartbeat from core client for 30 sec - exiting 17:51:43 (4716): No heartbeat from core client for 30 sec - exiting 17:51:44 (4716): No heartbeat from core client for 30 sec - exiting 17:51:45 (4716): No heartbeat from core client for 30 sec - exiting 17:51:46 (4716): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2764, selfPID=2764, iMonCtr=1 02:50:29 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:50:30 (5752): No heartbeat from core client for 30 sec - exiting 02:50:31 (5752): No heartbeat from core client for 30 sec - exiting 02:50:32 (5752): No heartbeat from core client for 30 sec - exiting 02:50:33 (5752): No heartbeat from core client for 30 sec - exiting 02:50:34 (5752): No heartbeat from core client for 30 sec - exiting 02:50:35 (5752): No heartbeat from core client for 30 sec - exiting 02:50:36 (5752): No heartbeat from core client for 30 sec - exiting 02:50:37 (5752): No heartbeat from core client for 30 sec - exiting 02:50:38 (5752): No heartbeat from core client for 30 sec - exiting 02:50:39 (5752): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5144, selfPID=5144, iMonCtr=1 13:49:31 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:49:32 (6012): No heartbeat from core client for 30 sec - exiting 13:49:33 (6012): No heartbeat from core client for 30 sec - exiting 13:49:34 (6012): No heartbeat from core client for 30 sec - exiting 13:49:35 (6012): No heartbeat from core client for 30 sec - exiting 13:49:36 (6012): No heartbeat from core client for 30 sec - exiting 13:49:37 (6012): No heartbeat from core client for 30 sec - exiting 13:49:38 (6012): No heartbeat from core client for 30 sec - exiting 13:49:39 (6012): No heartbeat from core client for 30 sec - exiting 13:49:40 (6012): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5860, selfPID=5860, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4492, selfPID=4492, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4112, selfPID=4112, iMonCtr=1 20:53:24 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:53:25 (3976): No heartbeat from core client for 30 sec - exiting 20:53:26 (3976): No heartbeat from core client for 30 sec - exiting 20:53:27 (3976): No heartbeat from core client for 30 sec - exiting 20:53:28 (3976): No heartbeat from core client for 30 sec - exiting 20:53:29 (3976): No heartbeat from core client for 30 sec - exiting 20:53:30 (3976): No heartbeat from core client for 30 sec - exiting 20:53:31 (3976): No heartbeat from core client for 30 sec - exiting 20:53:32 (3976): No heartbeat from core client for 30 sec - exiting 20:53:33 (3976): No heartbeat from core client for 30 sec - exiting 20:53:34 (3976): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4972, selfPID=4972, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3160, selfPID=3160, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3108, selfPID=3108, iMonCtr=1 01:00:21 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:22 (5620): No heartbeat from core client for 30 sec - exiting 01:00:23 (5620): No heartbeat from core client for 30 sec - exiting 01:00:24 (5620): No heartbeat from core client for 30 sec - exiting 01:00:25 (5620): No heartbeat from core client for 30 sec - exiting 01:00:26 (5620): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4788, selfPID=4788, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5456, selfPID=5456, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3784, selfPID=3784, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5764, selfPID=5764, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1524, selfPID=1524, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4236, selfPID=4236, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5920, selfPID=5920, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2696, selfPID=2696, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5740, selfPID=5740, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5608, selfPID=5608, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4944, selfPID=4944, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5932, selfPID=5932, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4428, selfPID=4428, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4420, selfPID=4420, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4660, selfPID=4660, iMonCtr=1 01:00:12 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:13 (3608): No heartbeat from core client for 30 sec - exiting 01:00:14 (3608): No heartbeat from core client for 30 sec - exiting 01:00:15 (3608): No heartbeat from core client for 30 sec - exiting 01:00:16 (3608): No heartbeat from core client for 30 sec - exiting 01:00:17 (3608): No heartbeat from core client for 30 sec - exiting 01:00:18 (3608): No heartbeat from core client for 30 sec - exiting 01:00:19 (3608): No heartbeat from core client for 30 sec - exiting 01:00:20 (3608): No heartbeat from core client for 30 sec - exiting 01:00:21 (3608): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4780, selfPID=4780, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5052, selfPID=5052, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2320, selfPID=2320, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4112, selfPID=4112, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3684, selfPID=3684, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3160, selfPID=3160, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1036, selfPID=1036, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1752, selfPID=1752, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4452, selfPID=4452, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=416, selfPID=416, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2536, selfPID=2536, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1812, selfPID=1812, iMonCtr=1 22:43:20 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:21 (5084): No heartbeat from core client for 30 sec - exiting 22:43:22 (5084): No heartbeat from core client for 30 sec - exiting 22:43:23 (5084): No heartbeat from core client for 30 sec - exiting 22:43:24 (5084): No heartbeat from core client for 30 sec - exiting 22:43:25 (5084): No heartbeat from core client for 30 sec - exiting 22:43:26 (5084): No heartbeat from core client for 30 sec - exiting 22:43:27 (5084): No heartbeat from core client for 30 sec - exiting 22:43:28 (5084): No heartbeat from core client for 30 sec - exiting 22:43:29 (5084): No heartbeat from core client for 30 sec - exiting 22:43:30 (5084): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2876, selfPID=2876, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3892, selfPID=3892, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3760, selfPID=3760, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3208, selfPID=3208, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2764, selfPID=2764, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2608, selfPID=2608, iMonCtr=1 17:19:26 (2432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:19:27 (2432): No heartbeat from core client for 30 sec - exiting 17:19:28 (2432): No heartbeat from core client for 30 sec - exiting 17:19:29 (2432): No heartbeat from core client for 30 sec - exiting 17:19:30 (2432): No heartbeat from core client for 30 sec - exiting 17:19:31 (2432): No heartbeat from core client for 30 sec - exiting 17:19:32 (2432): No heartbeat from core client for 30 sec - exiting 17:19:33 (2432): No heartbeat from core client for 30 sec - exiting 17:19:34 (2432): No heartbeat from core client for 30 sec - exiting 17:19:35 (2432): No heartbeat from core client for 30 sec - exiting 17:19:36 (2432): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3616, selfPID=3616, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4416, selfPID=4416, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3804, selfPID=3804, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4696, selfPID=4696, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4580, selfPID=4580, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4640, selfPID=4640, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4356, selfPID=4356, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4644, selfPID=4644, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3144, selfPID=3144, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4860, selfPID=4860, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4928, selfPID=4928, iMonCtr=1 23:18:25 (3056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:26 (3056): No heartbeat from core client for 30 sec - exiting 23:18:27 (3056): No heartbeat from core client for 30 sec - exiting 23:18:28 (3056): No heartbeat from core client for 30 sec - exiting 23:18:29 (3056): No heartbeat from core client for 30 sec - exiting 23:18:30 (3056): No heartbeat from core client for 30 sec - exiting 23:18:31 (3056): No heartbeat from core client for 30 sec - exiting 23:18:32 (3056): No heartbeat from core client for 30 sec - exiting 23:18:34 (3056): No heartbeat from core client for 30 sec - exiting 23:18:35 (3056): No heartbeat from core client for 30 sec - exiting 23:18:36 (3056): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1824, selfPID=1824, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4736, selfPID=4736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2768, selfPID=2768, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3556, selfPID=3556, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3660, selfPID=3660, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3324, selfPID=3324, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3224, selfPID=3224, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=456, selfPID=456, iMonCtr=1 01:00:03 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:04 (2148): No heartbeat from core client for 30 sec - exiting 01:00:05 (2148): No heartbeat from core client for 30 sec - exiting 01:00:06 (2148): No heartbeat from core client for 30 sec - exiting 01:00:07 (2148): No heartbeat from core client for 30 sec - exiting 01:00:08 (2148): No heartbeat from core client for 30 sec - exiting 01:00:09 (2148): No heartbeat from core client for 30 sec - exiting 01:00:10 (2148): No heartbeat from core client for 30 sec - exiting 01:00:11 (2148): No heartbeat from core client for 30 sec - exiting 01:00:12 (2148): No heartbeat from core client for 30 sec - exiting 01:00:13 (2148): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4880, selfPID=4880, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1400, selfPID=1400, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3888, selfPID=3888, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2148, selfPID=2148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4484, selfPID=4484, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=4648, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3244, selfPID=3244, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2944, selfPID=2944, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3148, selfPID=3148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4016, selfPID=4016, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4944, selfPID=4944, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4720, selfPID=4720, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1768, selfPID=1768, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=792, selfPID=792, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4628, selfPID=4628, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5052, selfPID=5052, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4284, selfPID=4284, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3676, selfPID=3676, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=852, selfPID=852, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2860, selfPID=2860, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4148, selfPID=4148, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3172, selfPID=3172, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2684, selfPID=2684, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4804, selfPID=4804, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4372, selfPID=4372, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1356, selfPID=1356, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3168, selfPID=3168, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2440, selfPID=2440, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4656, selfPID=4656, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4684, selfPID=4684, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4920, selfPID=4920, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4184, selfPID=4184, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2864, selfPID=2864, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4196, selfPID=4196, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3324, selfPID=3324, iMonCtr=1 02:25:29 (432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:30 (432): No heartbeat from core client for 30 sec - exiting 02:25:31 (432): No heartbeat from core client for 30 sec - exiting 02:25:32 (432): No heartbeat from core client for 30 sec - exiting 02:25:33 (432): No heartbeat from core client for 30 sec - exiting 02:25:34 (432): No heartbeat from core client for 30 sec - exiting 02:25:35 (432): No heartbeat from core client for 30 sec - exiting 02:25:36 (432): No heartbeat from core client for 30 sec - exiting 02:25:37 (432): No heartbeat from core client for 30 sec - exiting 02:25:38 (432): No heartbeat from core client for 30 sec - exiting 02:25:39 (432): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1864, selfPID=1864, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3644, selfPID=3644, iMonCtr=1 16:41:55 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:56 (4140): No heartbeat from core client for 30 sec - exiting 16:41:57 (4140): No heartbeat from core client for 30 sec - exiting 16:41:59 (4140): No heartbeat from core client for 30 sec - exiting 16:42:00 (4140): No heartbeat from core client for 30 sec - exiting 16:42:01 (4140): No heartbeat from core client for 30 sec - exiting 16:42:02 (4140): No heartbeat from core client for 30 sec - exiting 16:42:03 (4140): No heartbeat from core client for 30 sec - exiting 16:42:04 (4140): No heartbeat from core client for 30 sec - exiting 16:42:05 (4140): No heartbeat from core client for 30 sec - exiting 16:42:06 (4140): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4236, selfPID=4236, iMonCtr=1 Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2532, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jun 2011 20:26:31	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	959,040	2,550,725	2.6597
31 May 2011 22:30:37	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	933,120	2,496,237	2.6752
30 May 2011 15:15:57	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	907,200	2,442,010	2.6918
29 May 2011 17:52:17	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	881,280	2,388,618	2.7104
29 May 2011 01:50:44	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	855,360	2,333,587	2.7282
28 May 2011 11:25:23	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	829,440	2,281,736	2.7509
27 May 2011 20:55:32	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	803,520	2,229,635	2.7748
27 May 2011 06:29:03	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	777,600	2,177,577	2.8004
21 May 2011 02:00:47	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	751,680	2,115,771	2.8147
19 May 2011 22:29:42	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	725,760	2,050,814	2.8257
17 May 2011 10:45:55	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	699,840	1,987,079	2.8393
16 May 2011 03:19:22	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	673,920	1,913,729	2.8397
14 May 2011 13:44:08	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	648,000	1,843,937	2.8456
12 May 2011 15:37:55	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	622,080	1,768,679	2.8432
10 May 2011 07:55:28	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	596,160	1,693,391	2.8405
09 May 2011 08:05:50	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	570,240	1,631,030	2.8603
07 May 2011 19:13:24	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	544,320	1,565,246	2.8756
07 May 2011 03:28:52	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	518,400	1,509,785	2.9124
03 May 2011 15:58:11	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	492,480	1,447,419	2.9390
02 May 2011 14:49:15	1106381	12738690	hadcm3n_o2n0_1900_40_007198751_0	466,560	1,379,730	2.9572