Task 11920732

Name	hadsm3dhet2_u2ig_006726483_11
Workunit	6929826
Created	30 Sep 2010, 4:33:39 UTC
Sent	30 Sep 2010, 5:10:15 UTC
Report deadline	12 Sep 2011, 10:30:15 UTC
Received	20 Oct 2010, 15:01:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1042170
Run time	9 days 8 hours 21 min 18 sec
CPU time	8 days 18 hours 34 min 57 sec
Validate state	Invalid
Credit	4,664.44
Device peak FLOPS	2.51 GFLOPS
Application version	UK Met Office HadSM3 Slab Model v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN process is not running, exiting, bRetVal = 1, checkPID=6476, selfPID=6476, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5188, selfPID=5188, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=964, selfPID=964, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6928, selfPID=6928, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=560, selfPID=560, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6348, selfPID=6348, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5156, selfPID=5156, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5332, selfPID=5332, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN process is not running, exiting, bRetVal = 1, checkPID=616, selfPID=616, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2012, selfPID=2012, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4336, selfPID=4336, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6296, selfPID=6296, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=780, selfPID=780, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6644, selfPID=6644, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1756, selfPID=1756, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6192, selfPID=6192, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=368, selfPID=368, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3960, selfPID=3960, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5176, selfPID=5176, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4436, selfPID=4436, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1652, selfPID=1652, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6344, selfPID=6344, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2436, selfPID=2436, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=7080, selfPID=7080, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3928, selfPID=3928, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6328, selfPID=6328, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6452, selfPID=6452, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4276, selfPID=4276, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5796, selfPID=5796, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3828, selfPID=3828, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5752, selfPID=5752, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4944, selfPID=4944, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3516, selfPID=3516, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1776, selfPID=1776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4432, selfPID=4432, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5760, selfPID=5760, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1336, selfPID=1336, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3432, selfPID=3432, iMonCtr=1 MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. MainError: 10:00:41 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=776, selfPID=776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3244, selfPID=3244, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2768, selfPID=2768, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3432, selfPID=3432, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=7056, selfPID=7056, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2168, selfPID=2168, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6468, selfPID=6468, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN process is not running, exiting, bRetVal = 1, checkPID=6716, selfPID=6716, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5160, selfPID=5160, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6244, selfPID=6244, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5612, selfPID=5612, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2728, selfPID=2728, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=584, selfPID=584, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3716, selfPID=3716, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=7100, selfPID=7100, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5696, selfPID=5696, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5376, selfPID=5376, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5300, selfPID=5300, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4272, selfPID=4272, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2192, selfPID=2192, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1596, selfPID=1596, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6216, selfPID=6216, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6968, selfPID=6968, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6376, selfPID=6376, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6252, selfPID=6252, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5876, selfPID=5876, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6452, selfPID=6452, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5800, selfPID=5800, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1316, selfPID=1316, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5592, selfPID=5592, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6100, selfPID=6100, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1140, selfPID=1140, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3488, selfPID=3488, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6028, selfPID=6028, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2068, selfPID=2068, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5124, selfPID=5124, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5912, selfPID=5912, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2868, selfPID=2868, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6300, selfPID=6300, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=4460, selfPID=4460, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=6696, selfPID=6696, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1596, selfPID=1596, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=216, selfPID=216, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN process is not running, exiting, bRetVal = 1, checkPID=5908, selfPID=5908, iMonCtr=1 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5192, selfPID=5192, iMonCtr=1 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting CPDN process is not running, exiting, bRetVal = 1, checkPID=5444, selfPID=5444, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3872, selfPID=3872, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=3384, selfPID=3384, iMonCtr=1 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5328, selfPID=5328, iMonCtr=1 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN process is not running, exiting, bRetVal = 1, checkPID=5300, selfPID=5300, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2880, selfPID=2880, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=1516, selfPID=1516, iMonCtr=1 No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting No heartbeat from core client for 30 sec - exiting CPDN process is not running, exiting, bRetVal = 1, checkPID=5600, selfPID=5600, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=5956, selfPID=5956, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2612, selfPID=2612, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=976, selfPID=976, iMonCtr=1 CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=992, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( No Process Handle CPDN process is not running, exiting, bRetVal = 1, checkPID=2692, selfPID=2692, iMonCtr=1 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadsm3dhet2_u2ig_006726483/dataout/restart.day Model crashed: (null) Sorry, too many model crashes! :-( called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Oct 2010 08:39:57	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	248,446	728,151	1.4342
18 Oct 2010 22:55:29	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	237,644	712,690	1.4343
18 Oct 2010 18:16:41	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	226,842	697,410	1.4347
18 Oct 2010 13:25:50	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	216,040	682,179	1.4353
18 Oct 2010 08:55:12	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	205,238	666,741	1.4354
18 Oct 2010 03:55:23	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	194,436	651,442	1.4359
17 Oct 2010 23:12:10	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	183,634	636,307	1.4367
17 Oct 2010 18:27:53	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	172,832	621,131	1.4375
17 Oct 2010 13:37:21	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	162,030	605,839	1.4381
17 Oct 2010 03:35:21	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	151,228	590,544	1.4387
16 Oct 2010 19:00:16	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	140,426	574,805	1.4382
16 Oct 2010 13:12:38	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	129,624	559,336	1.4384
16 Oct 2010 04:41:58	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	118,822	543,868	1.4385
16 Oct 2010 03:17:10	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	108,020	528,363	1.4386
15 Oct 2010 19:46:53	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	97,218	512,923	1.4389
15 Oct 2010 09:51:17	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	86,416	497,545	1.4394
15 Oct 2010 03:29:12	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	75,614	482,016	1.4394
14 Oct 2010 20:21:17	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	64,812	466,293	1.4389
14 Oct 2010 15:46:24	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	54,010	450,682	1.4387
14 Oct 2010 08:53:20	1042170	11920732	hadsm3dhet2_u2ig_006726483_11	43,208	435,483	1.4398