Task 14732734

Name	hadcm3n_ydz2_1940_40_007858221_3
Workunit	8013333
Created	22 May 2012, 18:09:33 UTC
Sent	22 May 2012, 18:10:05 UTC
Report deadline	22 Aug 2012, 1:37:16 UTC
Received	27 Jul 2012, 8:43:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1219282
Run time	12 days 2 hours 59 min 22 sec
CPU time	10 days 0 hours 59 min 28 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4588, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4836, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3696, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1788, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4332, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1080, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3776, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:50:31 (1180): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:18:15 (3436): No heartbeat from core client for 30 sec - exiting 18:18:17 (3436): No heartbeat from core client for 30 sec - exiting 18:18:18 (3436): No heartbeat from core client for 30 sec - exiting 18:18:19 (3436): No heartbeat from core client for 30 sec - exiting 18:18:20 (3436): No heartbeat from core client for 30 sec - exiting 18:18:21 (3436): No heartbeat from core client for 30 sec - exiting 18:18:22 (3436): No heartbeat from core client for 30 sec - exiting 18:18:23 (3436): No heartbeat from core client for 30 sec - exiting 18:18:24 (3436): No heartbeat from core client for 30 sec - exiting 18:18:25 (3436): No heartbeat from core client for 30 sec - exiting 18:18:26 (3436): No heartbeat from core client for 30 sec - exiting 18:18:27 (3436): No heartbeat from core client for 30 sec - exiting 18:18:28 (3436): No heartbeat from core client for 30 sec - exiting 18:18:29 (3436): No heartbeat from core client for 30 sec - exiting 18:18:30 (3436): No heartbeat from core client for 30 sec - exiting 18:18:31 (3436): No heartbeat from core client for 30 sec - exiting 18:18:32 (3436): No heartbeat from core client for 30 sec - exiting 18:18:33 (3436): No heartbeat from core client for 30 sec - exiting 18:18:34 (3436): No heartbeat from core client for 30 sec - exiting 18:18:35 (3436): No heartbeat from core client for 30 sec - exiting 18:18:36 (3436): No heartbeat from core client for 30 sec - exiting 18:18:37 (3436): No heartbeat from core client for 30 sec - exiting 18:18:38 (3436): No heartbeat from core client for 30 sec - exiting 18:18:39 (3436): No heartbeat from core client for 30 sec - exiting 18:18:40 (3436): No heartbeat from core client for 30 sec - exiting 18:18:41 (3436): No heartbeat from core client for 30 sec - exiting 18:18:42 (3436): No heartbeat from core client for 30 sec - exiting 18:18:43 (3436): No heartbeat from core client for 30 sec - exiting 18:18:44 (3436): No heartbeat from core client for 30 sec - exiting 18:18:45 (3436): No heartbeat from core client for 30 sec - exiting 18:18:46 (3436): No heartbeat from core client for 30 sec - exiting 18:18:47 (3436): No heartbeat from core client for 30 sec - exiting 18:18:48 (3436): No heartbeat from core client for 30 sec - exiting 18:18:49 (3436): No heartbeat from core client for 30 sec - exiting 18:18:50 (3436): No heartbeat from core client for 30 sec - exiting 18:18:51 (3436): No heartbeat from core client for 30 sec - exiting 18:18:52 (3436): No heartbeat from core client for 30 sec - exiting 18:18:53 (3436): No heartbeat from core client for 30 sec - exiting 18:18:54 (3436): No heartbeat from core client for 30 sec - exiting 18:18:55 (3436): No heartbeat from core client for 30 sec - exiting 18:18:56 (3436): No heartbeat from core client for 30 sec - exiting 18:18:57 (3436): No heartbeat from core client for 30 sec - exiting 18:18:58 (3436): No heartbeat from core client for 30 sec - exiting 18:18:59 (3436): No heartbeat from core client for 30 sec - exiting 18:19:00 (3436): No heartbeat from core client for 30 sec - exiting 18:19:01 (3436): No heartbeat from core client for 30 sec - exiting 18:19:02 (3436): No heartbeat from core client for 30 sec - exiting 18:19:03 (3436): No heartbeat from core client for 30 sec - exiting 18:19:04 (3436): No heartbeat from core client for 30 sec - exiting 18:19:06 (3436): No heartbeat from core client for 30 sec - exiting 18:19:07 (3436): No heartbeat from core client for 30 sec - exiting 18:19:08 (3436): No heartbeat from core client for 30 sec - exiting 18:19:09 (3436): No heartbeat from core client for 30 sec - exiting 18:19:10 (3436): No heartbeat from core client for 30 sec - exiting 18:19:11 (3436): No heartbeat from core client for 30 sec - exiting 18:19:12 (3436): No heartbeat from core client for 30 sec - exiting 18:19:13 (3436): No heartbeat from core client for 30 sec - exiting 18:19:14 (3436): No heartbeat from core client for 30 sec - exiting 18:19:15 (3436): No heartbeat from core client for 30 sec - exiting 18:19:16 (3436): No heartbeat from core client for 30 sec - exiting 18:19:17 (3436): No heartbeat from core client for 30 sec - exiting 18:19:18 (3436): No heartbeat from core client for 30 sec - exiting 18:19:19 (3436): No heartbeat from core client for 30 sec - exiting 18:19:20 (3436): No heartbeat from core client for 30 sec - exiting 18:19:21 (3436): No heartbeat from core client for 30 sec - exiting 18:19:22 (3436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:08:06 (2808): No heartbeat from core client for 30 sec - exiting 15:08:07 (2808): No heartbeat from core client for 30 sec - exiting 15:08:08 (2808): No heartbeat from core client for 30 sec - exiting 15:08:09 (2808): No heartbeat from core client for 30 sec - exiting 15:08:10 (2808): No heartbeat from core client for 30 sec - exiting 15:08:11 (2808): No heartbeat from core client for 30 sec - exiting 15:08:12 (2808): No heartbeat from core client for 30 sec - exiting 15:08:13 (2808): No heartbeat from core client for 30 sec - exiting 15:08:14 (2808): No heartbeat from core client for 30 sec - exiting 15:08:15 (2808): No heartbeat from core client for 30 sec - exiting 15:08:16 (2808): No heartbeat from core client for 30 sec - exiting 15:08:17 (2808): No heartbeat from core client for 30 sec - exiting 15:08:18 (2808): No heartbeat from core client for 30 sec - exiting 15:08:19 (2808): No heartbeat from core client for 30 sec - exiting 15:08:20 (2808): No heartbeat from core client for 30 sec - exiting 15:08:21 (2808): No heartbeat from core client for 30 sec - exiting 15:08:22 (2808): No heartbeat from core client for 30 sec - exiting 15:08:23 (2808): No heartbeat from core client for 30 sec - exiting 15:08:24 (2808): No heartbeat from core client for 30 sec - exiting 15:08:25 (2808): No heartbeat from core client for 30 sec - exiting 15:08:26 (2808): No heartbeat from core client for 30 sec - exiting 15:08:27 (2808): No heartbeat from core client for 30 sec - exiting 15:08:28 (2808): No heartbeat from core client for 30 sec - exiting 15:08:29 (2808): No heartbeat from core client for 30 sec - exiting 15:08:30 (2808): No heartbeat from core client for 30 sec - exiting 15:08:31 (2808): No heartbeat from core client for 30 sec - exiting 15:08:32 (2808): No heartbeat from core client for 30 sec - exiting 15:08:33 (2808): No heartbeat from core client for 30 sec - exiting 15:08:34 (2808): No heartbeat from core client for 30 sec - exiting 15:08:35 (2808): No heartbeat from core client for 30 sec - exiting 15:08:36 (2808): No heartbeat from core client for 30 sec - exiting 15:08:37 (2808): No heartbeat from core client for 30 sec - exiting 15:08:38 (2808): No heartbeat from core client for 30 sec - exiting 15:08:39 (2808): No heartbeat from core client for 30 sec - exiting 15:08:40 (2808): No heartbeat from core client for 30 sec - exiting 15:08:41 (2808): No heartbeat from core client for 30 sec - exiting 15:08:42 (2808): No heartbeat from core client for 30 sec - exiting 15:08:43 (2808): No heartbeat from core client for 30 sec - exiting 15:08:44 (2808): No heartbeat from core client for 30 sec - exiting 15:08:45 (2808): No heartbeat from core client for 30 sec - exiting 15:08:46 (2808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:08:47 (2808): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6092, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2904, iMonCtr=1 Model crash detected, will try to restart... 07:43:42 (4592): No heartbeat from core client for 30 sec - exiting 07:43:43 (4592): No heartbeat from core client for 30 sec - exiting 07:43:44 (4592): No heartbeat from core client for 30 sec - exiting 07:43:45 (4592): No heartbeat from core client for 30 sec - exiting 07:43:46 (4592): No heartbeat from core client for 30 sec - exiting 07:43:47 (4592): No heartbeat from core client for 30 sec - exiting 07:43:48 (4592): No heartbeat from core client for 30 sec - exiting 07:43:49 (4592): No heartbeat from core client for 30 sec - exiting 07:43:50 (4592): No heartbeat from core client for 30 sec - exiting 07:43:51 (4592): No heartbeat from core client for 30 sec - exiting 07:43:52 (4592): No heartbeat from core client for 30 sec - exiting 07:43:53 (4592): No heartbeat from core client for 30 sec - exiting 07:43:54 (4592): No heartbeat from core client for 30 sec - exiting 07:43:55 (4592): No heartbeat from core client for 30 sec - exiting 07:43:56 (4592): No heartbeat from core client for 30 sec - exiting 07:43:57 (4592): No heartbeat from core client for 30 sec - exiting 07:43:58 (4592): No heartbeat from core client for 30 sec - exiting 07:43:59 (4592): No heartbeat from core client for 30 sec - exiting 07:44:00 (4592): No heartbeat from core client for 30 sec - exiting 07:44:01 (4592): No heartbeat from core client for 30 sec - exiting 07:44:02 (4592): No heartbeat from core client for 30 sec - exiting 07:44:03 (4592): No heartbeat from core client for 30 sec - exiting 07:44:04 (4592): No heartbeat from core client for 30 sec - exiting 07:44:05 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4164, iMonCtr=1 Model crash detected, will try to restart... 20:15:34 (788): No heartbeat from core client for 30 sec - exiting 20:15:35 (788): No heartbeat from core client for 30 sec - exiting 20:15:36 (788): No heartbeat from core client for 30 sec - exiting 20:15:37 (788): No heartbeat from core client for 30 sec - exiting 20:15:38 (788): No heartbeat from core client for 30 sec - exiting 20:15:39 (788): No heartbeat from core client for 30 sec - exiting 20:15:40 (788): No heartbeat from core client for 30 sec - exiting 20:15:41 (788): No heartbeat from core client for 30 sec - exiting 20:15:42 (788): No heartbeat from core client for 30 sec - exiting 20:15:44 (788): No heartbeat from core client for 30 sec - exiting 20:15:45 (788): No heartbeat from core client for 30 sec - exiting 20:15:46 (788): No heartbeat from core client for 30 sec - exiting 20:15:47 (788): No heartbeat from core client for 30 sec - exiting 20:15:48 (788): No heartbeat from core client for 30 sec - exiting 20:15:49 (788): No heartbeat from core client for 30 sec - exiting 20:15:50 (788): No heartbeat from core client for 30 sec - exiting 20:15:51 (788): No heartbeat from core client for 30 sec - exiting 20:15:52 (788): No heartbeat from core client for 30 sec - exiting 20:15:53 (788): No heartbeat from core client for 30 sec - exiting 20:15:55 (788): No heartbeat from core client for 30 sec - exiting 20:15:56 (788): No heartbeat from core client for 30 sec - exiting 20:15:57 (788): No heartbeat from core client for 30 sec - exiting 20:15:58 (788): No heartbeat from core client for 30 sec - exiting 20:15:59 (788): No heartbeat from core client for 30 sec - exiting 20:16:00 (788): No heartbeat from core client for 30 sec - exiting 20:16:01 (788): No heartbeat from core client for 30 sec - exiting 20:16:02 (788): No heartbeat from core client for 30 sec - exiting 20:16:03 (788): No heartbeat from core client for 30 sec - exiting 20:16:04 (788): No heartbeat from core client for 30 sec - exiting 20:16:05 (788): No heartbeat from core client for 30 sec - exiting 20:16:07 (788): No heartbeat from core client for 30 sec - exiting 20:16:08 (788): No heartbeat from core client for 30 sec - exiting 20:16:09 (788): No heartbeat from core client for 30 sec - exiting 20:16:10 (788): No heartbeat from core client for 30 sec - exiting 20:16:11 (788): No heartbeat from core client for 30 sec - exiting 20:16:12 (788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... 07:31:34 (4904): No heartbeat from core client for 30 sec - exiting 07:31:36 (4904): No heartbeat from core client for 30 sec - exiting 07:31:37 (4904): No heartbeat from core client for 30 sec - exiting 07:31:38 (4904): No heartbeat from core client for 30 sec - exiting 07:31:39 (4904): No heartbeat from core client for 30 sec - exiting 07:31:40 (4904): No heartbeat from core client for 30 sec - exiting 07:31:41 (4904): No heartbeat from core client for 30 sec - exiting 07:31:42 (4904): No heartbeat from core client for 30 sec - exiting 07:31:43 (4904): No heartbeat from core client for 30 sec - exiting 07:31:44 (4904): No heartbeat from core client for 30 sec - exiting 07:31:45 (4904): No heartbeat from core client for 30 sec - exiting 07:31:47 (4904): No heartbeat from core client for 30 sec - exiting 07:31:48 (4904): No heartbeat from core client for 30 sec - exiting 07:31:49 (4904): No heartbeat from core client for 30 sec - exiting 07:31:50 (4904): No heartbeat from core client for 30 sec - exiting 07:31:51 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3304, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4104, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:24:06 (5664): No heartbeat from core client for 30 sec - exiting 17:24:08 (5664): No heartbeat from core client for 30 sec - exiting 17:24:09 (5664): No heartbeat from core client for 30 sec - exiting 17:24:10 (5664): No heartbeat from core client for 30 sec - exiting 17:24:11 (5664): No heartbeat from core client for 30 sec - exiting 17:24:12 (5664): No heartbeat from core client for 30 sec - exiting 17:24:13 (5664): No heartbeat from core client for 30 sec - exiting 17:24:14 (5664): No heartbeat from core client for 30 sec - exiting 17:24:15 (5664): No heartbeat from core client for 30 sec - exiting 17:24:16 (5664): No heartbeat from core client for 30 sec - exiting 17:24:17 (5664): No heartbeat from core client for 30 sec - exiting 17:24:18 (5664): No heartbeat from core client for 30 sec - exiting 17:24:19 (5664): No heartbeat from core client for 30 sec - exiting 17:24:20 (5664): No heartbeat from core client for 30 sec - exiting 17:24:21 (5664): No heartbeat from core client for 30 sec - exiting 17:24:22 (5664): No heartbeat from core client for 30 sec - exiting 17:24:23 (5664): No heartbeat from core client for 30 sec - exiting 17:24:24 (5664): No heartbeat from core client for 30 sec - exiting 17:24:25 (5664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77A13F99 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ydz2_1940_40_007858221/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Jul 2012 08:45:10	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	518,400	867,337	1.6731
25 Jul 2012 17:42:45	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	492,480	825,979	1.6772
25 Jul 2012 04:41:24	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	466,560	783,816	1.6800
22 Jul 2012 21:45:00	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	440,640	741,239	1.6822
11 Jul 2012 20:43:33	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	414,720	699,221	1.6860
10 Jul 2012 03:14:21	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	388,800	655,915	1.6870
08 Jul 2012 18:19:55	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	362,880	611,772	1.6859
06 Jul 2012 12:39:53	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	336,960	568,747	1.6879
04 Jul 2012 22:46:35	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	311,040	523,896	1.6843
02 Jul 2012 21:15:50	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	285,120	478,189	1.6771
02 Jul 2012 12:00:59	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	259,200	434,669	1.6770
28 Jun 2012 12:12:09	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	233,280	390,582	1.6743
24 Jun 2012 20:07:08	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	207,360	346,753	1.6722
22 Jun 2012 19:31:53	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	181,440	301,912	1.6640
19 Jun 2012 23:19:00	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	155,520	259,126	1.6662
18 Jun 2012 18:56:10	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	129,600	215,714	1.6645
15 Jun 2012 04:10:11	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	103,680	172,591	1.6647
12 Jun 2012 23:26:49	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	77,760	129,067	1.6598
11 Jun 2012 04:41:21	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	51,840	85,488	1.6491
09 Jun 2012 20:19:29	1219282	14732734	hadcm3n_ydz2_1940_40_007858221_3	25,920	42,289	1.6315