Task 12778764

Name	hadam3p_pnw_z5fv_2002_1_006927971_1
Workunit	7131287
Created	7 Apr 2011, 6:28:01 UTC
Sent	7 Apr 2011, 7:58:14 UTC
Report deadline	19 Mar 2012, 13:18:14 UTC
Received	18 Jun 2011, 11:20:25 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1016212
Run time	17 days 23 hours 3 min 2 sec
CPU time	7 days 20 hours 25 min 15 sec
Validate state	Workunit error - check skipped
Credit	3,005.88
Device peak FLOPS	1.61 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Pacific North West v6.09 windows_intelx86
Stderr	<core_client_version>6.6.38</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5328, selfPID=5940, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3296, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5448, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5916, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=888, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5264, selfPID=5240, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2448, selfPID=5100, iMonCtr=1 Model crash detected, will try to restart... 16:05:21 (5228): No heartbeat from core client for 30 sec - exiting 16:05:22 (5228): No heartbeat from core client for 30 sec - exiting 16:05:23 (5228): No heartbeat from core client for 30 sec - exiting 16:05:24 (5228): No heartbeat from core client for 30 sec - exiting 16:05:25 (5228): No heartbeat from core client for 30 sec - exiting 16:05:26 (5228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:30 (2584): No heartbeat from core client for 30 sec - exiting 05:47:31 (2584): No heartbeat from core client for 30 sec - exiting 05:47:32 (2584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5896, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5424, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=2 Global Worker:: CPDN process is not running, ixiting,in ,RetVal = 1, check bRetVal =ID=0, selfPID=134 , iMonCtr=2 1, checkPID=0, selfPID=4384, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5116, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5688, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3008, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1560, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3000, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3524, selfPID=5532, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6100, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5584, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5480, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5432, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 1 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3132, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4628, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4336, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2744, selfPID=4732, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2020, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=912, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4248, selfPID=6032, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4480, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6136, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3264, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1564, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6124, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2340, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5760, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=332, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5120, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5136, iMonCtr=2 Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5828, selfPID=5444, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3088, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5912, selfPID=4324, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7884, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5140, selfPID=7548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4920, selfPID=6076, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4720, selfPID=1464, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 11:47:20 (5556): No heartbeat from core client for 30 sec - exiting 11:47:21 (5556): No heartbeat from core client for 30 sec - exiting 11:47:22 (5556): No heartbeat from core client for 30 sec - exiting 11:47:23 (5556): No heartbeat from core client for 30 sec - exiting 11:47:24 (5556): No heartbeat from core client for 30 sec - exiting 11:47:25 (5556): No heartbeat from core client for 30 sec - exiting 11:47:26 (5556): No heartbeat from core client for 30 sec - exiting 11:47:27 (5556): No heartbeat from core client for 30 sec - exiting 11:47:28 (5556): No heartbeat from core client for 30 sec - exiting 11:47:29 (5556): No heartbeat from core client for 30 sec - exiting 11:47:30 (5556): No heartbeat from core client for 30 sec - exiting 11:47:31 (5556): No heartbeat from core client for 30 sec - exiting 11:47:32 (5556): No heartbeat from core client for 30 sec - exiting 11:47:33 (5556): No heartbeat from core client for 30 sec - exiting 11:47:34 (5556): No heartbeat from core client for 30 sec - exiting 11:47:35 (5556): No heartbeat from core client for 30 sec - exiting 11:47:36 (5556): No heartbeat from core client for 30 sec - exiting 11:47:37 (5556): No heartbeat from core client for 30 sec - exiting 11:47:38 (5556): No heartbeat from core client for 30 sec - exiting 11:47:39 (5556): No heartbeat from core client for 30 sec - exiting 11:47:40 (5556): No heartbeat from core client for 30 sec - exiting 11:47:41 (5556): No heartbeat from core client for 30 sec - exiting 11:47:42 (5556): No heartbeat from core client for 30 sec - exiting 11:47:43 (5556): No heartbeat from core client for 30 sec - exiting 11:47:44 (5556): No heartbeat from core client for 30 sec - exiting 11:47:45 (5556): No heartbeat from core client for 30 sec - exiting 11:47:46 (5556): No heartbeat from core client for 30 sec - exiting 11:47:47 (5556): No heartbeat from core client for 30 sec - exiting 11:47:48 (5556): No heartbeat from core client for 30 sec - exiting 11:47:49 (5556): No heartbeat from core client for 30 sec - exiting 11:47:50 (5556): No heartbeat from core client for 30 sec - exiting 11:47:51 (5556): No heartbeat from core client for 30 sec - exiting 11:47:52 (5556): No heartbeat from core client for 30 sec - exiting 11:47:53 (5556): No heartbeat from core client for 30 sec - exiting 11:47:54 (5556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:41 (1040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:42 (1040): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5192, selfPID=5304, iMonCtr=1 Model crash detected, will try to restart... 12:31:24 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:31:25 (5896): No heartbeat from core client for 30 sec - exiting 12:31:26 (5896): No heartbeat from core client for 30 sec - exiting 12:31:27 (5896): No heartbeat from core client for 30 sec - exiting 12:31:28 (5896): No heartbeat from core client for 30 sec - exiting 12:31:29 (5896): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5184, selfPID=3772, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6100, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 09:27:55 (5556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:27:56 (5556): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8164, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 18:25:28 (1944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4720, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5860, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 3 CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2660, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2652, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 4 GCobao Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=2 2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2916, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5600, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 4 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2228, selfPID=224, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1068, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4604, selfPID=5764, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 4 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5456, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3220, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4396, selfPID=5748, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3836, selfPID=888, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 4 06:20:09 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:10 (5884): No heartbeat from core client for 30 sec - exiting 06:20:11 (5884): No heartbeat from core client for 30 sec - exiting 06:20:12 (5884): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3728, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CGntroller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2608, iMonCtr=2 Model crash detected, will try to restart... lobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=972, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3420, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4172, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3532, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5916, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2376, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5944, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4344, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7532, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7736, selfPID=5428, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr= 2 del crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5400, iMonCtr=2 Model crash detected, will try to restart... 06:26:26 (1156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:26:28 (1156): No heartbeat from core client for 30 sec - exiting 06:26:29 (1156): No heartbeat from core client for 30 sec - exiting 06:26:30 (1156): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5412, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4336, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2840, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4364, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6400, selfPID=8080, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6040, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6100, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5384, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 5 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5168, selfPID=4668, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5928, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=672, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5632, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6008, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7584, selfPID=5800, iMonCtr=1 Model crash detected, will try to restart... Colotroller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0al Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2820, iMonCtr=2 , selfPID=4256, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5508, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2168, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5432, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3680, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5588, selfPID=5996, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4656, selfPID=2208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4484, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 6 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6072, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5876, iMonCtr= 2 del crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr= 2 del crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3756, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5964, selfPID=2112, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1456, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6024, selfPID=5216, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6024, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 06:23:08 (4512): No heartbeat from core client for 30 sec - exiting 06:23:09 (4512): No heartbeat from core client for 30 sec - exiting 06:23:10 (4512): No heartbeat from core client for 30 sec - exiting 06:23:11 (4512): No heartbeat from core client for 30 sec - exiting 06:23:12 (4512): No heartbeat from core client for 30 sec - exiting 06:23:13 (4512): No heartbeat from core client for 30 sec - exiting 06:23:14 (4512): No heartbeat from core client for 30 sec - exiting 06:23:15 (4512): No heartbeat from core client for 30 sec - exiting 06:23:16 (4512): No heartbeat from core client for 30 sec - exiting 06:23:17 (4512): No heartbeat from core client for 30 sec - exiting 06:23:18 (4512): No heartbeat from core client for 30 sec - exiting 06:23:20 (4512): No heartbeat from core client for 30 sec - exiting 06:23:21 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CGntroller:: CPDN pr:cess is not running, exiting, b exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=2 RetVal = 1, checkPID=0, selfPID=3140, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 7 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6028, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6060, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4716, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4928, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=972, selfPID=3664, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2264, selfPID=4152, iMonCtr=1 Model crash detected, will try to restart... Glonobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5816, iMonCtr=2 odel crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=468, iMonCtr=2 CGlobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4804, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4136, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 22:22:22 (5480): No heartbeat from core client for 30 sec - exiting 22:22:23 (5480): No heartbeat from core client for 30 sec - exiting 22:22:24 (5480): No heartbeat from core client for 30 sec - exiting 22:22:25 (5480): No heartbeat from core client for 30 sec - exiting 22:22:26 (5480): No heartbeat from core client for 30 sec - exiting 22:22:27 (5480): No heartbeat from core client for 30 sec - exiting 22:22:28 (5480): No heartbeat from core client for 30 sec - exiting 22:22:29 (5480): No heartbeat from core client for 30 sec - exiting 22:22:30 (5480): No heartbeat from core client for 30 sec - exiting 22:22:31 (5480): No heartbeat from core client for 30 sec - exiting 22:22:32 (5480): No heartbeat from core client for 30 sec - exiting 22:22:33 (5480): No heartbeat from core client for 30 sec - exiting 22:22:34 (5480): No heartbeat from core client for 30 sec - exiting 22:22:35 (5480): No heartbeat from core client for 30 sec - exiting 22:22:36 (5480): No heartbeat from core client for 30 sec - exiting 22:22:37 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1468, selfPID=4956, iMonCtr=1 Model crash detected, will try to restart... 07:18:33 (336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:34 (336): No heartbeat from core client for 30 sec - exiting 07:18:35 (336): No heartbeat from core client for 30 sec - exiting 07:18:36 (336): No heartbeat from core client for 30 sec - exiting 07:18:37 (336): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1944, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5212, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4304, selfPID=2000, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2160, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3108, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 9 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3224, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5924, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5468, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 9 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2332, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 9 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1980, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4732, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4184, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4856, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 9 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5952, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 9 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1312, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6060, selfPID=4312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1212, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4508, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5224, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5968, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3860, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3348, selfPID=2856, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6692, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4000, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3740, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5984, iMonCtr= 2 del crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5960, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4456, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3092, selfPID=1400, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 10 CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4288, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 11 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4156, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 11 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5392, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 11 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6116, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2364, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 11 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3748, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6108, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 11 Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Jun 2011 22:03:44	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	138,336	677,246	4.8957
15 Jun 2011 06:11:03	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	126,835	620,964	4.8958
14 Jun 2011 13:23:30	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	126,816	620,109	4.8898
10 Jun 2011 06:21:01	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	115,298	562,941	4.8825
09 Jun 2011 10:56:19	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	115,296	562,193	4.8761
03 Jun 2011 06:15:28	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	103,776	507,324	4.8886
30 May 2011 08:44:00	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	92,256	455,029	4.9322
26 May 2011 05:19:30	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	80,738	399,250	4.9450
25 May 2011 06:42:57	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	80,736	398,521	4.9361
20 May 2011 05:24:01	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	69,224	344,678	4.9792
19 May 2011 13:07:13	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	69,220	344,008	4.9698
19 May 2011 07:37:52	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	69,216	343,316	4.9601
08 May 2011 06:31:45	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	57,696	288,199	4.9951
02 May 2011 10:00:33	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	46,176	230,994	5.0025
26 Apr 2011 08:13:26	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	34,656	174,463	5.0341
20 Apr 2011 15:31:08	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	23,136	115,907	5.0098
20 Apr 2011 15:31:08	1016212	12778764	hadam3p_pnw_z5fv_2002_1_006927971_1	11,616	58,382	5.0260