Task 16585387

Name	hadcm3n_88mm_1980_40_008720601_0
Workunit	8866579
Created	23 Apr 2014, 12:25:09 UTC
Sent	5 May 2014, 19:40:25 UTC
Report deadline	5 Aug 2014, 3:07:36 UTC
Received	7 Sep 2014, 15:55:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1326607
Run time	15 days 15 hours 26 min 5 sec
CPU time	15 days 9 hours 5 min 17 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:15:27 (3572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on 88mmko.dai4bu0 19:58:26 (3984): No heartbeat from core client for 30 sec - exiting 19:58:27 (3984): No heartbeat from core client for 30 sec - exiting 19:58:28 (3984): No heartbeat from core client for 30 sec - exiting 19:58:29 (3984): No heartbeat from core client for 30 sec - exiting 19:58:31 (3984): No heartbeat from core client for 30 sec - exiting 19:58:32 (3984): No heartbeat from core client for 30 sec - exiting 19:58:33 (3984): No heartbeat from core client for 30 sec - exiting 19:58:34 (3984): No heartbeat from core client for 30 sec - exiting 19:58:35 (3984): No heartbeat from core client for 30 sec - exiting 19:58:36 (3984): No heartbeat from core client for 30 sec - exiting 19:58:37 (3984): No heartbeat from core client for 30 sec - exiting 19:58:38 (3984): No heartbeat from core client for 30 sec - exiting 19:58:39 (3984): No heartbeat from core client for 30 sec - exiting 19:58:40 (3984): No heartbeat from core client for 30 sec - exiting 19:58:42 (3984): No heartbeat from core client for 30 sec - exiting 19:58:43 (3984): No heartbeat from core client for 30 sec - exiting 19:58:44 (3984): No heartbeat from core client for 30 sec - exiting 19:58:45 (3984): No heartbeat from core client for 30 sec - exiting 19:58:46 (3984): No heartbeat from core client for 30 sec - exiting 19:58:47 (3984): No heartbeat from core client for 30 sec - exiting 19:58:48 (3984): No heartbeat from core client for 30 sec - exiting 19:58:49 (3984): No heartbeat from core client for 30 sec - exiting 19:58:50 (3984): No heartbeat from core client for 30 sec - exiting 19:58:51 (3984): No heartbeat from core client for 30 sec - exiting 19:58:52 (3984): No heartbeat from core client for 30 sec - exiting 19:58:54 (3984): No heartbeat from core client for 30 sec - exiting 19:58:55 (3984): No heartbeat from core client for 30 sec - exiting 19:58:56 (3984): No heartbeat from core client for 30 sec - exiting 19:58:57 (3984): No heartbeat from core client for 30 sec - exiting 19:58:58 (3984): No heartbeat from core client for 30 sec - exiting 19:58:59 (3984): No heartbeat from core client for 30 sec - exiting 19:59:00 (3984): No heartbeat from core client for 30 sec - exiting 19:59:01 (3984): No heartbeat from core client for 30 sec - exiting 19:59:02 (3984): No heartbeat from core client for 30 sec - exiting 19:59:03 (3984): No heartbeat from core client for 30 sec - exiting 19:59:04 (3984): No heartbeat from core client for 30 sec - exiting 19:59:06 (3984): No heartbeat from core client for 30 sec - exiting 19:59:07 (3984): No heartbeat from core client for 30 sec - exiting 19:59:08 (3984): No heartbeat from core client for 30 sec - exiting 19:59:09 (3984): No heartbeat from core client for 30 sec - exiting 19:59:10 (3984): No heartbeat from core client for 30 sec - exiting 19:59:11 (3984): No heartbeat from core client for 30 sec - exiting 19:59:12 (3984): No heartbeat from core client for 30 sec - exiting 19:59:13 (3984): No heartbeat from core client for 30 sec - exiting 19:59:14 (3984): No heartbeat from core client for 30 sec - exiting 19:59:15 (3984): No heartbeat from core client for 30 sec - exiting 19:59:16 (3984): No heartbeat from core client for 30 sec - exiting 19:59:18 (3984): No heartbeat from core client for 30 sec - exiting 19:59:19 (3984): No heartbeat from core client for 30 sec - exiting 19:59:20 (3984): No heartbeat from core client for 30 sec - exiting 19:59:21 (3984): No heartbeat from core client for 30 sec - exiting 19:59:22 (3984): No heartbeat from core client for 30 sec - exiting 19:59:23 (3984): No heartbeat from core client for 30 sec - exiting 19:59:24 (3984): No heartbeat from core client for 30 sec - exiting 19:59:25 (3984): No heartbeat from core client for 30 sec - exiting 19:59:26 (3984): No heartbeat from core client for 30 sec - exiting 19:59:27 (3984): No heartbeat from core client for 30 sec - exiting 19:59:28 (3984): No heartbeat from core client for 30 sec - exiting 19:59:30 (3984): No heartbeat from core client for 30 sec - exiting 19:59:31 (3984): No heartbeat from core client for 30 sec - exiting 19:59:32 (3984): No heartbeat from core client for 30 sec - exiting 19:59:33 (3984): No heartbeat from core client for 30 sec - exiting 19:59:34 (3984): No heartbeat from core client for 30 sec - exiting 19:59:35 (3984): No heartbeat from core client for 30 sec - exiting 19:59:36 (3984): No heartbeat from core client for 30 sec - exiting 19:59:37 (3984): No heartbeat from core client for 30 sec - exiting 19:59:38 (3984): No heartbeat from core client for 30 sec - exiting 19:59:39 (3984): No heartbeat from core client for 30 sec - exiting 19:59:40 (3984): No heartbeat from core client for 30 sec - exiting 19:59:42 (3984): No heartbeat from core client for 30 sec - exiting 19:59:43 (3984): No heartbeat from core client for 30 sec - exiting 19:59:44 (3984): No heartbeat from core client for 30 sec - exiting 19:59:45 (3984): No heartbeat from core client for 30 sec - exiting 19:59:46 (3984): No heartbeat from core client for 30 sec - exiting 19:59:47 (3984): No heartbeat from core client for 30 sec - exiting 19:59:48 (3984): No heartbeat from core client for 30 sec - exiting 19:59:49 (3984): No heartbeat from core client for 30 sec - exiting 19:59:50 (3984): No heartbeat from core client for 30 sec - exiting 19:59:51 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4076, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:28:45 (3772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:43 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:19 (2764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... C05:51:11 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:54:42 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:54:43 (4008): No heartbeat from core client for 30 sec - exiting 22:54:45 (4008): No heartbeat from core client for 30 sec - exiting 22:54:46 (4008): No heartbeat from core client for 30 sec - exiting 22:54:47 (4008): No heartbeat from core client for 30 sec - exiting 22:54:48 (4008): No heartbeat from core client for 30 sec - exiting 22:54:49 (4008): No heartbeat from core client for 30 sec - exiting 22:54:50 (4008): No heartbeat from core client for 30 sec - exiting 22:54:51 (4008): No heartbeat from core client for 30 sec - exiting 22:54:52 (4008): No heartbeat from core client for 30 sec - exiting 22:54:53 (4008): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3652, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:10:00 (4128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1768, iMonCtr=1 Model crash detected, will try to restart... 13:41:25 (3672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:50:18 (3848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:56:36 (2260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77352AB1 read attempt to address 0x40BDBA0C Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77352AB1 read attempt to address 0x40BDBA0C Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_88mm_1980_40_008720601/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Sep 2014 13:20:11	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	1,036,800	1,328,193	1.2811
04 Sep 2014 19:39:21	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	1,010,880	1,294,500	1.2806
30 Aug 2014 20:41:06	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	984,960	1,261,206	1.2805
30 Aug 2014 10:54:50	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	959,040	1,229,326	1.2818
30 Aug 2014 05:51:15	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	933,120	1,196,631	1.2824
23 Aug 2014 20:15:31	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	907,200	1,163,358	1.2824
19 Aug 2014 18:30:01	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	881,280	1,129,448	1.2816
07 Aug 2014 21:29:09	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	855,360	1,096,779	1.2822
07 Aug 2014 21:29:09	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	829,440	1,064,073	1.2829
07 Aug 2014 21:29:09	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	803,520	1,030,726	1.2828
07 Aug 2014 21:29:09	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	777,600	996,984	1.2821
07 Aug 2014 21:29:09	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	751,680	964,226	1.2828
01 Aug 2014 07:17:21	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	725,760	930,506	1.2821
28 Jul 2014 13:17:45	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	699,840	898,147	1.2834
28 Jul 2014 04:56:36	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	673,920	865,981	1.2850
27 Jul 2014 19:17:43	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	648,000	833,838	1.2868
24 Jul 2014 16:21:41	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	622,080	799,825	1.2857
20 Jul 2014 16:52:43	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	596,160	767,254	1.2870
05 Jul 2014 04:29:24	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	570,240	734,311	1.2877
03 Jul 2014 20:28:16	1326607	16585387	hadcm3n_88mm_1980_40_008720601_0	544,320	701,959	1.2896