 |
|
| Name | hadcm3n_t67t_1940_40_007448431_1 |
| Workunit | 7645934 |
| Created | 9 Sep 2011, 21:05:23 UTC |
| Sent | 15 Sep 2011, 8:25:57 UTC |
| Report deadline | 15 Dec 2011, 15:53:08 UTC |
| Received | 23 Nov 2011, 17:45:05 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 193 (0x000000C1) EXIT_SIGNAL |
| Computer ID | 1103925 |
| Run time | 11 days 7 hours 23 min 27 sec |
| CPU time | 10 days 20 hours 51 min 14 sec |
| Validate state | Invalid |
| Credit | 6,220.80 |
| Device peak FLOPS | 1.72 GFLOPS |
| Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu |
| Stderr | <core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 11:38:32 (6973): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:34 (6973): No heartbeat from core client for 30 sec - exiting 19:55:36 (11228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:35:51 (1465): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:34:34 (1469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:25 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:50:00 (1606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:19 (9204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:38 (10021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:56:33 (11327): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:47 (13121): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:49 (13121): No heartbeat from core client for 30 sec - exiting SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb78ad400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x805721f] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x806e448] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837738c] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7723bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15824, iMonCtr=1 Model crash detected, will try to restart... 17:41:18 (1430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:41 (4578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:26:44 (1666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:53 (1666): No heartbeat from core client for 30 sec - exiting 08:26:54 (1666): No heartbeat from core client for 30 sec - exiting 08:26:55 (1666): No heartbeat from core client for 30 sec - exiting 08:26:56 (1666): No heartbeat from core client for 30 sec - exiting 08:26:57 (1666): No heartbeat from core client for 30 sec - exiting 08:26:58 (1666): No heartbeat from core client for 30 sec - exiting 18:11:09 (7878): No heartbeat from core client for 30 sec - exiting 18:11:14 (7878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:15 (7878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:22:48 (20758): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:35 (22605): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:56 (22605): No heartbeat from core client for 30 sec - exiting 13:30:02 (26399): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:13 (382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:24 (382): No heartbeat from core client for 30 sec - exiting 13:13:07 (4066): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:30 (4066): No heartbeat from core client for 30 sec - exiting 13:13:31 (4066): No heartbeat from core client for 30 sec - exiting 13:13:32 (4066): No heartbeat from core client for 30 sec - exiting 13:13:33 (4066): No heartbeat from core client for 30 sec - exiting 13:13:34 (4066): No heartbeat from core client for 30 sec - exiting 13:13:35 (4066): No heartbeat from core client for 30 sec - exiting 13:13:36 (4066): No heartbeat from core client for 30 sec - exiting 13:13:37 (4066): No heartbeat from core client for 30 sec - exiting 13:13:40 (4066): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:31:14 (9660): No heartbeat from core client for 30 sec - exiting 14:31:22 (9660): No heartbeat from core client for 30 sec - exiting 14:31:23 (9660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:31:24 (9660): No heartbeat from core client for 30 sec - exiting 14:31:25 (9660): No heartbeat from core client for 30 sec - exiting 14:31:26 (9660): No heartbeat from core client for 30 sec - exiting 14:31:27 (9660): No heartbeat from core client for 30 sec - exiting 14:52:37 (10773): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:45:34 (11741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:45:57 (11741): No heartbeat from core client for 30 sec - exiting 21:45:58 (11741): No heartbeat from core client for 30 sec - exiting 08:43:13 (14340): No heartbeat from core client for 30 sec - exiting 08:43:14 (14340): No heartbeat from core client for 30 sec - exiting 08:43:15 (14340): No heartbeat from core client for 30 sec - exiting 08:43:16 (14340): No heartbeat from core client for 30 sec - exiting 08:43:17 (14340): No heartbeat from core client for 30 sec - exiting 08:43:18 (14340): No heartbeat from core client for 30 sec - exiting 08:43:19 (14340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:20 (14340): No heartbeat from core client for 30 sec - exiting 13:44:29 (18187): No heartbeat from core client for 30 sec - exiting 13:44:36 (18187): No heartbeat from core client for 30 sec - exiting 13:44:37 (18187): No heartbeat from core client for 30 sec - exiting 13:44:38 (18187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:21 (20642): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:17 (23419): No heartbeat from core client for 30 sec - exiting 09:04:18 (23419): No heartbeat from core client for 30 sec - exiting 09:04:19 (23419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:55 (23430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:14 (23430): No heartbeat from core client for 30 sec - exiting 19:12:15 (23430): No heartbeat from core client for 30 sec - exiting 19:12:16 (23430): No heartbeat from core client for 30 sec - exiting 19:12:17 (23430): No heartbeat from core client for 30 sec - exiting 19:12:18 (23430): No heartbeat from core client for 30 sec - exiting 08:31:56 (25966): No heartbeat from core client for 30 sec - exiting 08:32:08 (25966): No heartbeat from core client for 30 sec - exiting 08:32:09 (25966): No heartbeat from core client for 30 sec - exiting 08:32:10 (25966): No heartbeat from core client for 30 sec - exiting 08:32:11 (25966): No heartbeat from core client for 30 sec - exiting 08:32:12 (25966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:14 (25966): No heartbeat from core client for 30 sec - exiting 08:32:15 (25966): No heartbeat from core client for 30 sec - exiting 08:32:16 (25966): No heartbeat from core client for 30 sec - exiting 08:32:17 (25966): No heartbeat from core client for 30 sec - exiting 08:32:18 (25966): No heartbeat from core client for 30 sec - exiting 20:14:12 (28784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:18 (28784): No heartbeat from core client for 30 sec - exiting 21:33:41 (31688): No heartbeat from core client for 30 sec - exiting 21:33:57 (31688): No heartbeat from core client for 30 sec - exiting 21:33:58 (31688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:00 (31688): No heartbeat from core client for 30 sec - exiting 21:34:01 (31688): No heartbeat from core client for 30 sec - exiting 21:34:02 (31688): No heartbeat from core client for 30 sec - exiting 21:34:03 (31688): No heartbeat from core client for 30 sec - exiting 21:34:05 (31688): No heartbeat from core client for 30 sec - exiting 21:34:06 (31688): No heartbeat from core client for 30 sec - exiting 21:34:07 (31688): No heartbeat from core client for 30 sec - exiting 21:34:08 (31688): No heartbeat from core client for 30 sec - exiting 21:34:11 (31688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:18:42 (1449): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:58:05 (4005): No heartbeat from core client for 30 sec - exiting 13:58:16 (4005): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:50:34 (8983): No heartbeat from core client for 30 sec - exiting 20:51:03 (8983): No heartbeat from core client for 30 sec - exiting 20:51:04 (8983): No heartbeat from core client for 30 sec - exiting 20:51:05 (8983): No heartbeat from core client for 30 sec - exiting 20:51:06 (8983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:12:36 (1459): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:19:18 (9509): No heartbeat from core client for 30 sec - exiting 17:19:33 (9509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:21:41 (11377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:40 (23149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:45 (23149): No heartbeat from core client for 30 sec - exiting 11:08:46 (23149): No heartbeat from core client for 30 sec - exiting 11:08:47 (23149): No heartbeat from core client for 30 sec - exiting 13:37:17 (31029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:26 (31029): No heartbeat from core client for 30 sec - exiting 18:42:29 (429): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:10:29 (1793): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:01:10 (10907): No heartbeat from core client for 30 sec - exiting 10:01:43 (10907): No heartbeat from core client for 30 sec - exiting 10:01:44 (10907): No heartbeat from core client for 30 sec - exiting 10:01:45 (10907): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:46 (10907): No heartbeat from core client for 30 sec - exiting 10:01:47 (10907): No heartbeat from core client for 30 sec - exiting 10:01:49 (10907): No heartbeat from core client for 30 sec - exiting 10:01:50 (10907): No heartbeat from core client for 30 sec - exiting 15:00:36 (15784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:38 (15784): No heartbeat from core client for 30 sec - exiting 15:00:39 (15784): No heartbeat from core client for 30 sec - exiting 15:00:40 (15784): No heartbeat from core client for 30 sec - exiting 15:00:41 (15784): No heartbeat from core client for 30 sec - exiting 15:00:42 (15784): No heartbeat from core client for 30 sec - exiting *** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08b2b0d8 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0xb7532591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0xb7533de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0xb7536ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xb7721741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xb772179d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74ddbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:01 36439625 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 36439625 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08ad5000-08b3b000 rw-p 00000000 00:00 0 [heap] b6f00000-b6f21000 rw-p 00000000 00:00 0 b6f21000-b7000000 ---p 00000000 00:00 0 b704a000-b74c5000 rw-s 00000000 08:01 36438689 /var/lib/boinc-client/slots/0/134610 b74c5000-b74c7000 rw-p 00000000 00:00 0 b74c7000-b761a000 r-xp 00000000 08:01 22678211 /lib/tls/i686/cmov/libc-2.11.1.so b761a000-b761b000 ---p 00153000 08:01 22678211 /lib/tls/i686/cmov/libc-2.11.1.so b761b000-b761d000 r--p 00153000 08:01 22678211 /lib/tls/i686/cmov/libc-2.11.1.so b761d000-b761e000 rw-p 00155000 08:01 22678211 /lib/tls/i686/cmov/libc-2.11.1.so b761e000-b7621000 rw-p 00000000 00:00 0 b7621000-b763e000 r-xp 00000000 08:01 22544467 /lib/libgcc_s.so.1 b763e000-b763f000 r--p 0001c000 08:01 22544467 /lib/libgcc_s.so.1 b763f000-b7640000 rw-p 0001d000 08:01 22544467 /lib/libgcc_s.so.1 b7640000-b7664000 r-xp 00000000 08:01 22678215 /lib/tls/i686/cmov/libm-2.11.1.so b7664000-b7665000 r--p 00023000 08:01 22678215 /lib/tls/i686/cmov/libm-2.11.1.so b7665000-b7666000 rw-p 00024000 08:01 22678215 /lib/tls/i686/cmov/libm-2.11.1.so b7666000-b774f000 r-xp 00000000 08:01 59117235 /usr/lib/libstdc++.so.6.0.13 b774f000-b7750000 ---p 000e9000 08:01 59117235 /usr/lib/libstdc++.so.6.0.13 b7750000-b7754000 r--p 000e9000 08:01 59117235 /usr/lib/libstdc++.so.6.0.13 b7754000-b7755000 rw-p 000ed000 08:01 59117235 /usr/lib/libstdc++.so.6.0.13 b7755000-b775d000 rw-p 00000000 00:00 0 b775d000-b775f000 r-xp 00000000 08:01 22678214 /lib/tls/i686/cmov/libdl-2.11.1.so b775f000-b7760000 r--p 00001000 08:01 22678214 /lib/tls/i686/cmov/libdl-2.11.1.so b7760000-b7761000 rw-p 00002000 08:01 22678214 /lib/tls/i686/cmov/libdl-2.11.1.so b7761000-b7776000 r-xp 00000000 08:01 22678227 /lib/tls/i686/cmov/libpthread-2.11.1.so b7776000-b7777000 r--p 00014000 08:01 22678227 /lib/tls/i686/cmov/libpthread-2.11.1.so b7777000-b7778000 rw-p 00015000 08:01 22678227 /lib/tls/i686/cmov/libpthread-2.11.1.so b7778000-b777a000 rw-p 00000000 00:00 0 b7789000-b778a000 rw-p 00000000 00:00 0 b778a000-b778b000 ---p 00000000 00:00 0 b778b000-b778e000 rw-p 00000000 00:00 0 b778e000-b7790000 rw-s 00000000 08:01 36442233 /var/lib/boinc-client/slots/0/boinc_mmap_file b7790000-b7792000 rw-p 00000000 00:00 0 b7792000-b7793000 r-xp 00000000 00:00 0 [vdso] b7793000-b77ae000 r-xp 00000000 08:01 22544444 /lib/ld-2.11.1.so b77ae000-b77af000 r--p 0001a000 08:01 22544444 /lib/ld-2.11.1.so b77af000-b77b0000 rw-p 0001b000 08:01 22544444 /lib/ld-2.11.1.so bfa2f000-bfa9f000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7792400] [0xb7792430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb74f1651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb74f4a82] /lib/tls/i686/cmov/libc.so.6(+0x6149d)[0xb752849d] /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0xb7532591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0xb7533de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0xb7536ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0xb7721741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0xb772179d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb74ddbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 21 Nov 2011 17:05:43 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 518,400 | 939,070 | 1.8115 |
| 16 Nov 2011 19:39:02 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 492,480 | 892,487 | 1.8122 |
| 16 Nov 2011 19:39:02 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 466,560 | 846,203 | 1.8137 |
| 08 Nov 2011 16:49:57 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 440,640 | 798,775 | 1.8128 |
| 07 Nov 2011 19:13:31 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 414,720 | 752,608 | 1.8147 |
| 03 Nov 2011 17:57:14 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 388,800 | 707,168 | 1.8188 |
| 02 Nov 2011 09:39:16 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 362,880 | 660,754 | 1.8209 |
| 31 Oct 2011 19:30:20 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 336,960 | 614,013 | 1.8222 |
| 31 Oct 2011 18:08:45 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 311,040 | 567,854 | 1.8257 |
| 31 Oct 2011 15:06:03 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 285,120 | 521,720 | 1.8298 |
| 31 Oct 2011 15:06:03 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 259,200 | 475,201 | 1.8333 |
| 18 Oct 2011 13:12:58 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 233,280 | 428,453 | 1.8366 |
| 17 Oct 2011 09:47:05 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 207,360 | 381,641 | 1.8405 |
| 12 Oct 2011 13:44:46 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 181,440 | 333,793 | 1.8397 |
| 07 Oct 2011 17:24:05 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 155,520 | 286,601 | 1.8429 |
| 03 Oct 2011 19:38:26 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 129,600 | 239,035 | 1.8444 |
| 30 Sep 2011 19:48:47 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 103,680 | 190,143 | 1.8339 |
| 26 Sep 2011 00:14:03 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 77,760 | 142,904 | 1.8378 |
| 25 Sep 2011 00:40:18 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 51,840 | 95,621 | 1.8445 |
| 16 Sep 2011 22:55:20 | 1103925 | 13361414 | hadcm3n_t67t_1940_40_007448431_1 | 25,920 | 47,572 | 1.8353 |
©2024 climateprediction.net