Task 13616664

Name	hadcm3n_o18b_1980_40_007537703_3
Workunit	7734935
Created	7 Nov 2011, 20:50:22 UTC
Sent	7 Nov 2011, 21:08:59 UTC
Report deadline	7 Feb 2012, 4:36:10 UTC
Received	6 Feb 2012, 19:28:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1178822
Run time	49 days 8 hours 34 min 19 sec
CPU time	47 days 20 hours 52 min
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	0.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 10:52:43 (1556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:24 (5347): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:29 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:15 (5405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:34:47 (5455): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:55:56 (1194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:40 (1194): No heartbeat from core client for 30 sec - exiting 21:57:41 (1194): No heartbeat from core client for 30 sec - exiting 21:57:43 (1194): No heartbeat from core client for 30 sec - exiting 21:57:44 (1194): No heartbeat from core client for 30 sec - exiting 21:57:45 (1194): No heartbeat from core client for 30 sec - exiting 21:57:50 (1194): No heartbeat from core client for 30 sec - exiting 21:58:02 (1194): No heartbeat from core client for 30 sec - exiting 21:58:03 (1194): No heartbeat from core client for 30 sec - exiting 21:58:04 (1194): No heartbeat from core client for 30 sec - exiting 21:58:06 (1194): No heartbeat from core client for 30 sec - exiting 21:58:07 (1194): No heartbeat from core client for 30 sec - exiting 21:58:08 (1194): No heartbeat from core client for 30 sec - exiting 21:58:09 (1194): No heartbeat from core client for 30 sec - exiting 21:58:10 (1194): No heartbeat from core client for 30 sec - exiting 21:58:11 (1194): No heartbeat from core client for 30 sec - exiting 21:58:12 (1194): No heartbeat from core client for 30 sec - exiting 21:58:13 (1194): No heartbeat from core client for 30 sec - exiting 21:58:14 (1194): No heartbeat from core client for 30 sec - exiting 21:58:21 (1194): No heartbeat from core client for 30 sec - exiting 21:58:22 (1194): No heartbeat from core client for 30 sec - exiting 21:58:23 (1194): No heartbeat from core client for 30 sec - exiting 21:58:24 (1194): No heartbeat from core client for 30 sec - exiting 21:58:25 (1194): No heartbeat from core client for 30 sec - exiting 21:58:26 (1194): No heartbeat from core client for 30 sec - exiting 21:58:27 (1194): No heartbeat from core client for 30 sec - exiting 21:58:28 (1194): No heartbeat from core client for 30 sec - exiting 21:58:29 (1194): No heartbeat from core client for 30 sec - exiting 21:58:30 (1194): No heartbeat from core client for 30 sec - exiting 21:58:31 (1194): No heartbeat from core client for 30 sec - exiting 21:58:32 (1194): No heartbeat from core client for 30 sec - exiting 21:58:33 (1194): No heartbeat from core client for 30 sec - exiting 21:58:34 (1194): No heartbeat from core client for 30 sec - exiting 21:58:35 (1194): No heartbeat from core client for 30 sec - exiting 21:58:36 (1194): No heartbeat from core client for 30 sec - exiting 21:58:37 (1194): No heartbeat from core client for 30 sec - exiting 21:58:38 (1194): No heartbeat from core client for 30 sec - exiting 21:58:39 (1194): No heartbeat from core client for 30 sec - exiting 21:58:40 (1194): No heartbeat from core client for 30 sec - exiting 21:58:41 (1194): No heartbeat from core client for 30 sec - exiting 21:58:42 (1194): No heartbeat from core client for 30 sec - exiting 21:58:43 (1194): No heartbeat from core client for 30 sec - exiting 21:58:44 (1194): No heartbeat from core client for 30 sec - exiting 21:58:45 (1194): No heartbeat from core client for 30 sec - exiting 21:58:46 (1194): No heartbeat from core client for 30 sec - exiting 21:58:47 (1194): No heartbeat from core client for 30 sec - exiting 21:58:48 (1194): No heartbeat from core client for 30 sec - exiting 21:58:49 (1194): No heartbeat from core client for 30 sec - exiting 21:58:50 (1194): No heartbeat from core client for 30 sec - exiting 21:58:51 (1194): No heartbeat from core client for 30 sec - exiting 21:58:52 (1194): No heartbeat from core client for 30 sec - exiting 21:58:53 (1194): No heartbeat from core client for 30 sec - exiting 21:58:54 (1194): No heartbeat from core client for 30 sec - exiting 21:59:06 (1194): No heartbeat from core client for 30 sec - exiting 21:59:07 (1194): No heartbeat from core client for 30 sec - exiting 21:59:08 (1194): No heartbeat from core client for 30 sec - exiting 21:59:09 (1194): No heartbeat from core client for 30 sec - exiting 21:59:10 (1194): No heartbeat from core client for 30 sec - exiting 21:59:11 (1194): No heartbeat from core client for 30 sec - exiting 21:59:12 (1194): No heartbeat from core client for 30 sec - exiting 21:59:24 (1194): No heartbeat from core client for 30 sec - exiting 21:59:25 (1194): No heartbeat from core client for 30 sec - exiting 21:59:26 (1194): No heartbeat from core client for 30 sec - exiting 21:59:27 (1194): No heartbeat from core client for 30 sec - exiting 21:59:28 (1194): No heartbeat from core client for 30 sec - exiting 21:59:29 (1194): No heartbeat from core client for 30 sec - exiting 21:59:30 (1194): No heartbeat from core client for 30 sec - exiting 21:59:31 (1194): No heartbeat from core client for 30 sec - exiting 21:59:52 (1194): No heartbeat from core client for 30 sec - exiting 21:59:53 (1194): No heartbeat from core client for 30 sec - exiting 21:59:57 (1194): No heartbeat from core client for 30 sec - exiting 21:59:58 (1194): No heartbeat from core client for 30 sec - exiting 21:59:59 (1194): No heartbeat from core client for 30 sec - exiting 22:00:00 (1194): No heartbeat from core client for 30 sec - exiting 22:00:01 (1194): No heartbeat from core client for 30 sec - exiting 22:00:02 (1194): No heartbeat from core client for 30 sec - exiting 22:00:03 (1194): No heartbeat from core client for 30 sec - exiting 22:00:10 (1194): No heartbeat from core client for 30 sec - exiting 22:00:11 (1194): No heartbeat from core client for 30 sec - exiting 22:00:12 (1194): No heartbeat from core client for 30 sec - exiting 22:00:13 (1194): No heartbeat from core client for 30 sec - exiting 22:00:14 (1194): No heartbeat from core client for 30 sec - exiting 22:00:15 (1194): No heartbeat from core client for 30 sec - exiting 22:00:16 (1194): No heartbeat from core client for 30 sec - exiting 22:00:17 (1194): No heartbeat from core client for 30 sec - exiting 22:00:18 (1194): No heartbeat from core client for 30 sec - exiting 22:00:47 (1194): No heartbeat from core client for 30 sec - exiting 22:00:48 (1194): No heartbeat from core client for 30 sec - exiting 22:00:57 (1194): No heartbeat from core client for 30 sec - exiting 22:00:58 (1194): No heartbeat from core client for 30 sec - exiting 22:00:59 (1194): No heartbeat from core client for 30 sec - exiting 22:01:00 (1194): No heartbeat from core client for 30 sec - exiting 22:01:37 (1194): No heartbeat from core client for 30 sec - exiting 22:01:38 (1194): No heartbeat from core client for 30 sec - exiting 22:01:39 (1194): No heartbeat from core client for 30 sec - exiting 22:01:40 (1194): No heartbeat from core client for 30 sec - exiting 22:01:41 (1194): No heartbeat from core client for 30 sec - exiting 22:01:42 (1194): No heartbeat from core client for 30 sec - exiting 22:01:43 (1194): No heartbeat from core client for 30 sec - exiting 22:01:44 (1194): No heartbeat from core client for 30 sec - exiting 22:01:45 (1194): No heartbeat from core client for 30 sec - exiting 22:01:46 (1194): No heartbeat from core client for 30 sec - exiting 22:01:47 (1194): No heartbeat from core client for 30 sec - exiting 22:01:48 (1194): No heartbeat from core client for 30 sec - exiting 22:01:49 (1194): No heartbeat from core client for 30 sec - exiting 22:01:50 (1194): No heartbeat from core client for 30 sec - exiting 22:01:51 (1194): No heartbeat from core client for 30 sec - exiting 22:01:52 (1194): No heartbeat from core client for 30 sec - exiting 22:01:53 (1194): No heartbeat from core client for 30 sec - exiting 22:01:54 (1194): No heartbeat from core client for 30 sec - exiting 22:02:25 (1194): No heartbeat from core client for 30 sec - exiting 22:02:26 (1194): No heartbeat from core client for 30 sec - exiting 22:02:40 (1194): No heartbeat from core client for 30 sec - exiting 22:02:43 (1194): No heartbeat from core client for 30 sec - exiting 22:02:44 (1194): No heartbeat from core client for 30 sec - exiting 22:02:45 (1194): No heartbeat from core client for 30 sec - exiting 22:02:46 (1194): No heartbeat from core client for 30 sec - exiting 22:02:47 (1194): No heartbeat from core client for 30 sec - exiting 22:02:48 (1194): No heartbeat from core client for 30 sec - exiting 22:02:49 (1194): No heartbeat from core client for 30 sec - exiting 22:02:50 (1194): No heartbeat from core client for 30 sec - exiting 22:02:51 (1194): No heartbeat from core client for 30 sec - exiting 22:03:07 (1194): No heartbeat from core client for 30 sec - exiting 22:03:08 (1194): No heartbeat from core client for 30 sec - exiting 22:03:09 (1194): No heartbeat from core client for 30 sec - exiting 22:03:10 (1194): No heartbeat from core client for 30 sec - exiting 22:03:11 (1194): No heartbeat from core client for 30 sec - exiting 22:03:19 (1194): No heartbeat from core client for 30 sec - exiting 22:03:20 (1194): No heartbeat from core client for 30 sec - exiting 22:03:21 (1194): No heartbeat from core client for 30 sec - exiting 22:03:22 (1194): No heartbeat from core client for 30 sec - exiting 22:03:24 (1194): No heartbeat from core client for 30 sec - exiting 22:03:25 (1194): No heartbeat from core client for 30 sec - exiting 22:03:26 (1194): No heartbeat from core client for 30 sec - exiting 22:03:27 (1194): No heartbeat from core client for 30 sec - exiting 22:03:30 (1194): No heartbeat from core client for 30 sec - exiting 22:03:31 (1194): No heartbeat from core client for 30 sec - exiting 22:03:32 (1194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 09:54:47 (1628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... 19:56:16 (1486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:03 (1723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:05 (1733): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:48 (1743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:50 (1753): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:33 (1764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:35 (1774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:18 (1785): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:20 (1795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:04:03 (1805): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:15 (1816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:06:38 (1826): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:45 (1836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:28 (1846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:12:36 (1856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:24 (1866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:06 (1876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:18 (1886): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:17:20 (1896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:19 (1910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:07 (1920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:05 (1930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:14 (1941): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 3 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (9 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x42d400] /usr/lib/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsgetnEPci+0x47)[0x37f247] /usr/lib/libstdc++.so.6(_ZNSi4readEPci+0x74)[0x357ca4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804e44f] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050f96] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x14cbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Feb 2012 20:44:49	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	803,520	4,057,105	5.0492
28 Jan 2012 21:45:37	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	777,600	3,930,117	5.0542
24 Jan 2012 09:39:19	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	751,680	3,798,875	5.0538
20 Jan 2012 22:37:59	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	725,760	3,667,309	5.0531
14 Jan 2012 01:35:36	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	699,840	3,536,596	5.0534
10 Jan 2012 03:57:17	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	673,920	3,405,112	5.0527
08 Jan 2012 02:13:34	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	648,000	3,273,920	5.0523
05 Jan 2012 17:01:41	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	622,080	3,142,805	5.0521
04 Jan 2012 02:54:44	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	596,160	3,011,915	5.0522
01 Jan 2012 20:16:20	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	570,240	2,881,603	5.0533
30 Dec 2011 13:32:27	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	544,320	2,750,328	5.0528
28 Dec 2011 23:36:07	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	518,400	2,619,512	5.0531
22 Dec 2011 04:17:04	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	492,480	2,488,035	5.0521
18 Dec 2011 07:27:16	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	466,560	2,356,985	5.0518
16 Dec 2011 18:06:33	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	440,640	2,226,124	5.0520
15 Dec 2011 05:18:25	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	414,720	2,095,747	5.0534
13 Dec 2011 16:23:27	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	388,800	1,965,002	5.0540
12 Dec 2011 03:21:59	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	362,880	1,834,307	5.0549
10 Dec 2011 01:19:16	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	336,960	1,702,673	5.0530
08 Dec 2011 12:04:02	1178822	13616664	hadcm3n_o18b_1980_40_007537703_3	311,040	1,570,924	5.0506