climateprediction.net home page
Task 17806711

Task 17806711

Name hadcm3n_xbmu_1940_40_009151372_3
Workunit 9281708
Created 19 Jan 2015, 2:16:50 UTC
Sent 19 Jan 2015, 4:20:43 UTC
Report deadline 20 Apr 2015, 11:47:54 UTC
Received 17 Oct 2016, 14:40:54 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1332638
Run time 8 days 20 hours 48 min 50 sec
CPU time 8 days 8 hours 16 min 26 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 3.30 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Peak disk usage 75.75 MB
Stderr
<core_client_version>7.6.33</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
r 30 sec - exiting
22:14:33 (5289): No heartbeat from core client for 30 sec - exiting
22:14:34 (5289): No heartbeat from core client for 30 sec - exiting
22:14:40 (5289): No heartbeat from core client for 30 sec - exiting
22:14:41 (5289): No heartbeat from core client for 30 sec - exiting
22:14:42 (5289): No heartbeat from core client for 30 sec - exiting
22:14:43 (5289): No heartbeat from core client for 30 sec - exiting
22:14:44 (5289): No heartbeat from core client for 30 sec - exiting
22:14:45 (5289): No heartbeat from core client for 30 sec - exiting
22:14:46 (5289): No heartbeat from core client for 30 sec - exiting
22:14:47 (5289): No heartbeat from core client for 30 sec - exiting
22:14:48 (5289): No heartbeat from core client for 30 sec - exiting
22:14:49 (5289): No heartbeat from core client for 30 sec - exiting
21:34:34 (5289): No heartbeat from core client for 30 sec - exiting
21:34:35 (5289): No heartbeat from core client for 30 sec - exiting
04:47:37 (5289): No heartbeat from core client for 30 sec - exiting
04:47:38 (5289): No heartbeat from core client for 30 sec - exiting
04:47:39 (5289): No heartbeat from core client for 30 sec - exiting
04:47:40 (5289): No heartbeat from core client for 30 sec - exiting
04:47:41 (5289): No heartbeat from core client for 30 sec - exiting
07:04:41 (5289): No heartbeat from core client for 30 sec - exiting
07:04:42 (5289): No heartbeat from core client for 30 sec - exiting
07:04:43 (5289): No heartbeat from core client for 30 sec - exiting
07:04:44 (5289): No heartbeat from core client for 30 sec - exiting
07:04:45 (5289): No heartbeat from core client for 30 sec - exiting
07:04:46 (5289): No heartbeat from core client for 30 sec - exiting
07:04:47 (5289): No heartbeat from core client for 30 sec - exiting
07:04:48 (5289): No heartbeat from core client for 30 sec - exiting
07:04:49 (5289): No heartbeat from core client for 30 sec - exiting
07:04:50 (5289): No heartbeat from core client for 30 sec - exiting
07:04:51 (5289): No heartbeat from core client for 30 sec - exiting
07:04:52 (5289): No heartbeat from core client for 30 sec - exiting
07:04:53 (5289): No heartbeat from core client for 30 sec - exiting
07:04:54 (5289): No heartbeat from core client for 30 sec - exiting
07:04:55 (5289): No heartbeat from core client for 30 sec - exiting
07:04:56 (5289): No heartbeat from core client for 30 sec - exiting
10:15:55 (5289): No heartbeat from core client for 30 sec - exiting
10:15:56 (5289): No heartbeat from core client for 30 sec - exiting
10:15:57 (5289): No heartbeat from core client for 30 sec - exiting
12:02:44 (5289): No heartbeat from core client for 30 sec - exiting
12:02:45 (5289): No heartbeat from core client for 30 sec - exiting
12:02:46 (5289): No heartbeat from core client for 30 sec - exiting
12:02:47 (5289): No heartbeat from core client for 30 sec - exiting
12:02:48 (5289): No heartbeat from core client for 30 sec - exiting
12:02:49 (5289): No heartbeat from core client for 30 sec - exiting
12:02:50 (5289): No heartbeat from core client for 30 sec - exiting
12:02:51 (5289): No heartbeat from core client for 30 sec - exiting
12:02:52 (5289): No heartbeat from core client for 30 sec - exiting
12:02:53 (5289): No heartbeat from core client for 30 sec - exiting
12:02:54 (5289): No heartbeat from core client for 30 sec - exiting
12:02:55 (5289): No heartbeat from core client for 30 sec - exiting
12:02:56 (5289): No heartbeat from core client for 30 sec - exiting
12:02:57 (5289): No heartbeat from core client for 30 sec - exiting
12:02:58 (5289): No heartbeat from core client for 30 sec - exiting
17:53:32 (5289): No heartbeat from core client for 30 sec - exiting
17:53:33 (5289): No heartbeat from core client for 30 sec - exiting
17:53:34 (5289): No heartbeat from core client for 30 sec - exiting
21:49:56 (5289): No heartbeat from core client for 30 sec - exiting
21:49:59 (5289): No heartbeat from core client for 30 sec - exiting
21:50:00 (5289): No heartbeat from core client for 30 sec - exiting
21:50:01 (5289): No heartbeat from core client for 30 sec - exiting
21:50:02 (5289): No heartbeat from core client for 30 sec - exiting
21:50:03 (5289): No heartbeat from core client for 30 sec - exiting
21:50:04 (5289): No heartbeat from core client for 30 sec - exiting
21:50:05 (5289): No heartbeat from core client for 30 sec - exiting
21:50:06 (5289): No heartbeat from core client for 30 sec - exiting
21:50:07 (5289): No heartbeat from core client for 30 sec - exiting
21:50:08 (5289): No heartbeat from core client for 30 sec - exiting
21:50:09 (5289): No heartbeat from core client for 30 sec - exiting
21:50:10 (5289): No heartbeat from core client for 30 sec - exiting
21:50:11 (5289): No heartbeat from core client for 30 sec - exiting
21:50:12 (5289): No heartbeat from core client for 30 sec - exiting
21:50:13 (5289): No heartbeat from core client for 30 sec - exiting
21:50:14 (5289): No heartbeat from core client for 30 sec - exiting
21:50:15 (5289): No heartbeat from core client for 30 sec - exiting
21:50:16 (5289): No heartbeat from core client for 30 sec - exiting
21:50:17 (5289): No heartbeat from core client for 30 sec - exiting
21:50:18 (5289): No heartbeat from core client for 30 sec - exiting
21:50:19 (5289): No heartbeat from core client for 30 sec - exiting
21:50:20 (5289): No heartbeat from core client for 30 sec - exiting
21:50:21 (5289): No heartbeat from core client for 30 sec - exiting
21:50:22 (5289): No heartbeat from core client for 30 sec - exiting
21:50:23 (5289): No heartbeat from core client for 30 sec - exiting
21:50:24 (5289): No heartbeat from core client for 30 sec - exiting
21:50:25 (5289): No heartbeat from core client for 30 sec - exiting
21:50:26 (5289): No heartbeat from core client for 30 sec - exiting
21:50:27 (5289): No heartbeat from core client for 30 sec - exiting
21:50:28 (5289): No heartbeat from core client for 30 sec - exiting
21:50:29 (5289): No heartbeat from core client for 30 sec - exiting
21:50:30 (5289): No heartbeat from core client for 30 sec - exiting
21:50:31 (5289): No heartbeat from core client for 30 sec - exiting
21:50:32 (5289): No heartbeat from core client for 30 sec - exiting
21:50:33 (5289): No heartbeat from core client for 30 sec - exiting
21:50:34 (5289): No heartbeat from core client for 30 sec - exiting
21:50:35 (5289): No heartbeat from core client for 30 sec - exiting
21:50:36 (5289): No heartbeat from core client for 30 sec - exiting
21:50:37 (5289): No heartbeat from core client for 30 sec - exiting
21:50:38 (5289): No heartbeat from core client for 30 sec - exiting
21:50:39 (5289): No heartbeat from core client for 30 sec - exiting
21:50:40 (5289): No heartbeat from core client for 30 sec - exiting
21:50:41 (5289): No heartbeat from core client for 30 sec - exiting
21:50:42 (5289): No heartbeat from core client for 30 sec - exiting
21:50:43 (5289): No heartbeat from core client for 30 sec - exiting
21:50:44 (5289): No heartbeat from core client for 30 sec - exiting
21:50:45 (5289): No heartbeat from core client for 30 sec - exiting
21:50:46 (5289): No heartbeat from core client for 30 sec - exiting
21:50:47 (5289): No heartbeat from core client for 30 sec - exiting
21:50:48 (5289): No heartbeat from core client for 30 sec - exiting
21:50:49 (5289): No heartbeat from core client for 30 sec - exiting
21:50:50 (5289): No heartbeat from core client for 30 sec - exiting
21:50:51 (5289): No heartbeat from core client for 30 sec - exiting
21:50:52 (5289): No heartbeat from core client for 30 sec - exiting
21:50:53 (5289): No heartbeat from core client for 30 sec - exiting
21:50:54 (5289): No heartbeat from core client for 30 sec - exiting
21:50:55 (5289): No heartbeat from core client for 30 sec - exiting
21:50:56 (5289): No heartbeat from core client for 30 sec - exiting
21:50:57 (5289): No heartbeat from core client for 30 sec - exiting
21:50:58 (5289): No heartbeat from core client for 30 sec - exiting
21:50:59 (5289): No heartbeat from core client for 30 sec - exiting
21:51:00 (5289): No heartbeat from core client for 30 sec - exiting
21:51:01 (5289): No heartbeat from core client for 30 sec - exiting
21:51:10 (5289): No heartbeat from core client for 30 sec - exiting
21:51:11 (5289): No heartbeat from core client for 30 sec - exiting
21:51:12 (5289): No heartbeat from core client for 30 sec - exiting
21:51:13 (5289): No heartbeat from core client for 30 sec - exiting
21:51:14 (5289): No heartbeat from core client for 30 sec - exiting
07:33:29 (5289): No heartbeat from core client for 30 sec - exiting
07:33:31 (5289): No heartbeat from core client for 30 sec - exiting
07:33:32 (5289): No heartbeat from core client for 30 sec - exiting
08:51:59 (5289): No heartbeat from core client for 30 sec - exiting
08:52:00 (5289): No heartbeat from core client for 30 sec - exiting
08:52:01 (5289): No heartbeat from core client for 30 sec - exiting
08:52:02 (5289): No heartbeat from core client for 30 sec - exiting
08:52:03 (5289): No heartbeat from core client for 30 sec - exiting
08:52:04 (5289): No heartbeat from core client for 30 sec - exiting
08:52:05 (5289): No heartbeat from core client for 30 sec - exiting
08:52:06 (5289): No heartbeat from core client for 30 sec - exiting
08:52:07 (5289): No heartbeat from core client for 30 sec - exiting
08:52:08 (5289): No heartbeat from core client for 30 sec - exiting
23:37:42 (5289): No heartbeat from core client for 30 sec - exiting
23:37:43 (5289): No heartbeat from core client for 30 sec - exiting
23:37:44 (5289): No heartbeat from core client for 30 sec - exiting
23:37:45 (5289): No heartbeat from core client for 30 sec - exiting
23:37:46 (5289): No heartbeat from core client for 30 sec - exiting
23:37:47 (5289): No heartbeat from core client for 30 sec - exiting
23:37:48 (5289): No heartbeat from core client for 30 sec - exiting
23:37:49 (5289): No heartbeat from core client for 30 sec - exiting
23:37:50 (5289): No heartbeat from core client for 30 sec - exiting
23:37:51 (5289): No heartbeat from core client for 30 sec - exiting
23:37:52 (5289): No heartbeat from core client for 30 sec - exiting
23:37:53 (5289): No heartbeat from core client for 30 sec - exiting
23:37:54 (5289): No heartbeat from core client for 30 sec - exiting
23:37:55 (5289): No heartbeat from core client for 30 sec - exiting
23:37:56 (5289): No heartbeat from core client for 30 sec - exiting
23:37:57 (5289): No heartbeat from core client for 30 sec - exiting
23:37:58 (5289): No heartbeat from core client for 30 sec - exiting
23:37:59 (5289): No heartbeat from core client for 30 sec - exiting
23:38:00 (5289): No heartbeat from core client for 30 sec - exiting
23:38:01 (5289): No heartbeat from core client for 30 sec - exiting
23:38:02 (5289): No heartbeat from core client for 30 sec - exiting
23:38:03 (5289): No heartbeat from core client for 30 sec - exiting
23:38:04 (5289): No heartbeat from core client for 30 sec - exiting
23:38:05 (5289): No heartbeat from core client for 30 sec - exiting
23:38:06 (5289): No heartbeat from core client for 30 sec - exiting
23:38:07 (5289): No heartbeat from core client for 30 sec - exiting
23:38:08 (5289): No heartbeat from core client for 30 sec - exiting
23:38:09 (5289): No heartbeat from core client for 30 sec - exiting
23:38:10 (5289): No heartbeat from core client for 30 sec - exiting
23:38:11 (5289): No heartbeat from core client for 30 sec - exiting
23:38:12 (5289): No heartbeat from core client for 30 sec - exiting
23:38:13 (5289): No heartbeat from core client for 30 sec - exiting
13:22:45 (5289): No heartbeat from core client for 30 sec - exiting
13:22:46 (5289): No heartbeat from core client for 30 sec - exiting
13:22:47 (5289): No heartbeat from core client for 30 sec - exiting
13:22:48 (5289): No heartbeat from core client for 30 sec - exiting
13:22:49 (5289): No heartbeat from core client for 30 sec - exiting
13:22:50 (5289): No heartbeat from core client for 30 sec - exiting
13:22:51 (5289): No heartbeat from core client for 30 sec - exiting
13:22:52 (5289): No heartbeat from core client for 30 sec - exiting
13:22:53 (5289): No heartbeat from core client for 30 sec - exiting
13:22:54 (5289): No heartbeat from core client for 30 sec - exiting
16:37:58 (5289): No heartbeat from core client for 30 sec - exiting
16:37:59 (5289): No heartbeat from core client for 30 sec - exiting
16:38:00 (5289): No heartbeat from core client for 30 sec - exiting
16:38:01 (5289): No heartbeat from core client for 30 sec - exiting
16:38:02 (5289): No heartbeat from core client for 30 sec - exiting
16:38:03 (5289): No heartbeat from core client for 30 sec - exiting
19:17:15 (5289): No heartbeat from core client for 30 sec - exiting
19:17:16 (5289): No heartbeat from core client for 30 sec - exiting
20:45:09 (5289): No heartbeat from core client for 30 sec - exiting
20:45:10 (5289): No heartbeat from core client for 30 sec - exiting
00:45:23 (5289): No heartbeat from core client for 30 sec - exiting
00:45:24 (5289): No heartbeat from core client for 30 sec - exiting
00:45:25 (5289): No heartbeat from core client for 30 sec - exiting
00:45:26 (5289): No heartbeat from core client for 30 sec - exiting
02:30:26 (5289): No heartbeat from core client for 30 sec - exiting
09:20:49 (5289): No heartbeat from core client for 30 sec - exiting
09:20:51 (5289): No heartbeat from core client for 30 sec - exiting
09:20:52 (5289): No heartbeat from core client for 30 sec - exiting
09:20:53 (5289): No heartbeat from core client for 30 sec - exiting
09:20:54 (5289): No heartbeat from core client for 30 sec - exiting
09:20:55 (5289): No heartbeat from core client for 30 sec - exiting
09:20:56 (5289): No heartbeat from core client for 30 sec - exiting
14:06:04 (5289): No heartbeat from core client for 30 sec - exiting
14:06:05 (5289): No heartbeat from core client for 30 sec - exiting
14:06:07 (5289): No heartbeat from core client for 30 sec - exiting
14:06:08 (5289): No heartbeat from core client for 30 sec - exiting
14:06:09 (5289): No heartbeat from core client for 30 sec - exiting
19:42:01 (5289): No heartbeat from core client for 30 sec - exiting
13:41:14 (5289): No heartbeat from core client for 30 sec - exiting
13:41:15 (5289): No heartbeat from core client for 30 sec - exiting
13:41:16 (5289): No heartbeat from core client for 30 sec - exiting
07:50:37 (5289): No heartbeat from core client for 30 sec - exiting
07:50:39 (5289): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x093979f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
21:45:36 (25623): No heartbeat from core client for 30 sec - exiting
21:45:37 (25623): No heartbeat from core client for 30 sec - exiting
21:45:38 (25623): No heartbeat from core client for 30 sec - exiting
21:45:39 (25623): No heartbeat from core client for 30 sec - exiting
21:45:40 (25623): No heartbeat from core client for 30 sec - exiting
21:45:41 (25623): No heartbeat from core client for 30 sec - exiting
21:45:42 (25623): No heartbeat from core client for 30 sec - exiting
21:45:43 (25623): No heartbeat from core client for 30 sec - exiting
21:45:44 (25623): No heartbeat from core client for 30 sec - exiting
21:45:45 (25623): No heartbeat from core client for 30 sec - exiting
18:45:12 (25623): No heartbeat from core client for 30 sec - exiting
07:51:34 (25623): No heartbeat from core client for 30 sec - exiting
20:28:56 (25623): No heartbeat from core client for 30 sec - exiting
20:28:58 (25623): No heartbeat from core client for 30 sec - exiting
20:28:59 (25623): No heartbeat from core client for 30 sec - exiting
20:29:00 (25623): No heartbeat from core client for 30 sec - exiting
20:29:01 (25623): No heartbeat from core client for 30 sec - exiting
20:29:02 (25623): No heartbeat from core client for 30 sec - exiting
09:14:32 (25623): No heartbeat from core client for 30 sec - exiting
09:14:34 (25623): No heartbeat from core client for 30 sec - exiting
12:40:16 (25623): No heartbeat from core client for 30 sec - exiting
12:40:17 (25623): No heartbeat from core client for 30 sec - exiting
12:40:18 (25623): No heartbeat from core client for 30 sec - exiting
12:40:19 (25623): No heartbeat from core client for 30 sec - exiting
12:40:20 (25623): No heartbeat from core client for 30 sec - exiting
12:40:21 (25623): No heartbeat from core client for 30 sec - exiting
10:20:52 (25623): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x096819f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
18:27:21 (4275): No heartbeat from core client for 30 sec - exiting
18:27:22 (4275): No heartbeat from core client for 30 sec - exiting
18:27:23 (4275): No heartbeat from core client for 30 sec - exiting
18:27:24 (4275): No heartbeat from core client for 30 sec - exiting
18:27:25 (4275): No heartbeat from core client for 30 sec - exiting
18:27:26 (4275): No heartbeat from core client for 30 sec - exiting
18:27:27 (4275): No heartbeat from core client for 30 sec - exiting
18:27:28 (4275): No heartbeat from core client for 30 sec - exiting
18:27:29 (4275): No heartbeat from core client for 30 sec - exiting
18:27:30 (4275): No heartbeat from core client for 30 sec - exiting
18:27:31 (4275): No heartbeat from core client for 30 sec - exiting
18:27:32 (4275): No heartbeat from core client for 30 sec - exiting
18:27:33 (4275): No heartbeat from core client for 30 sec - exiting
18:27:34 (4275): No heartbeat from core client for 30 sec - exiting
18:27:35 (4275): No heartbeat from core client for 30 sec - exiting
18:27:36 (4275): No heartbeat from core client for 30 sec - exiting
18:27:37 (4275): No heartbeat from core client for 30 sec - exiting
18:27:38 (4275): No heartbeat from core client for 30 sec - exiting
18:27:39 (4275): No heartbeat from core client for 30 sec - exiting
18:27:40 (4275): No heartbeat from core client for 30 sec - exiting
18:27:41 (4275): No heartbeat from core client for 30 sec - exiting
18:27:42 (4275): No heartbeat from core client for 30 sec - exiting
18:27:43 (4275): No heartbeat from core client for 30 sec - exiting
18:27:44 (4275): No heartbeat from core client for 30 sec - exiting
18:27:45 (4275): No heartbeat from core client for 30 sec - exiting
18:27:46 (4275): No heartbeat from core client for 30 sec - exiting
18:27:47 (4275): No heartbeat from core client for 30 sec - exiting
18:27:48 (4275): No heartbeat from core client for 30 sec - exiting
18:27:49 (4275): No heartbeat from core client for 30 sec - exiting
18:27:50 (4275): No heartbeat from core client for 30 sec - exiting
18:27:51 (4275): No heartbeat from core client for 30 sec - exiting
18:27:52 (4275): No heartbeat from core client for 30 sec - exiting
18:27:53 (4275): No heartbeat from core client for 30 sec - exiting
18:27:54 (4275): No heartbeat from core client for 30 sec - exiting
18:27:55 (4275): No heartbeat from core client for 30 sec - exiting
18:27:56 (4275): No heartbeat from core client for 30 sec - exiting
18:27:57 (4275): No heartbeat from core client for 30 sec - exiting
18:27:58 (4275): No heartbeat from core client for 30 sec - exiting
18:27:59 (4275): No heartbeat from core client for 30 sec - exiting
18:28:00 (4275): No heartbeat from core client for 30 sec - exiting
18:28:01 (4275): No heartbeat from core client for 30 sec - exiting
18:28:02 (4275): No heartbeat from core client for 30 sec - exiting
18:28:03 (4275): No heartbeat from core client for 30 sec - exiting
18:28:04 (4275): No heartbeat from core client for 30 sec - exiting
18:28:05 (4275): No heartbeat from core client for 30 sec - exiting
18:28:06 (4275): No heartbeat from core client for 30 sec - exiting
18:28:07 (4275): No heartbeat from core client for 30 sec - exiting
18:28:08 (4275): No heartbeat from core client for 30 sec - exiting
18:28:09 (4275): No heartbeat from core client for 30 sec - exiting
18:28:10 (4275): No heartbeat from core client for 30 sec - exiting
18:28:14 (4275): No heartbeat from core client for 30 sec - exiting
18:28:15 (4275): No heartbeat from core client for 30 sec - exiting
18:28:16 (4275): No heartbeat from core client for 30 sec - exiting
18:28:17 (4275): No heartbeat from core client for 30 sec - exiting
18:28:18 (4275): No heartbeat from core client for 30 sec - exiting
18:28:19 (4275): No heartbeat from core client for 30 sec - exiting
18:28:20 (4275): No heartbeat from core client for 30 sec - exiting
18:28:21 (4275): No heartbeat from core client for 30 sec - exiting
18:28:22 (4275): No heartbeat from core client for 30 sec - exiting
18:28:23 (4275): No heartbeat from core client for 30 sec - exiting
18:28:24 (4275): No heartbeat from core client for 30 sec - exiting
18:28:25 (4275): No heartbeat from core client for 30 sec - exiting
18:28:26 (4275): No heartbeat from core client for 30 sec - exiting
18:28:27 (4275): No heartbeat from core client for 30 sec - exiting
18:28:28 (4275): No heartbeat from core client for 30 sec - exiting
18:28:29 (4275): No heartbeat from core client for 30 sec - exiting
18:28:30 (4275): No heartbeat from core client for 30 sec - exiting
18:28:31 (4275): No heartbeat from core client for 30 sec - exiting
18:28:32 (4275): No heartbeat from core client for 30 sec - exiting
09:47:53 (4275): No heartbeat from core client for 30 sec - exiting
09:47:55 (4275): No heartbeat from core client for 30 sec - exiting
09:47:56 (4275): No heartbeat from core client for 30 sec - exiting
09:47:57 (4275): No heartbeat from core client for 30 sec - exiting
09:47:58 (4275): No heartbeat from core client for 30 sec - exiting
09:47:59 (4275): No heartbeat from core client for 30 sec - exiting
22:05:11 (4275): No heartbeat from core client for 30 sec - exiting
22:05:12 (4275): No heartbeat from core client for 30 sec - exiting
22:05:13 (4275): No heartbeat from core client for 30 sec - exiting
09:12:30 (4275): No heartbeat from core client for 30 sec - exiting
09:12:33 (4275): No heartbeat from core client for 30 sec - exiting
09:12:34 (4275): No heartbeat from core client for 30 sec - exiting
09:12:35 (4275): No heartbeat from core client for 30 sec - exiting
09:12:36 (4275): No heartbeat from core client for 30 sec - exiting
09:12:37 (4275): No heartbeat from core client for 30 sec - exiting
09:12:38 (4275): No heartbeat from core client for 30 sec - exiting
09:12:39 (4275): No heartbeat from core client for 30 sec - exiting
09:12:40 (4275): No heartbeat from core client for 30 sec - exiting
09:12:41 (4275): No heartbeat from core client for 30 sec - exiting
09:12:42 (4275): No heartbeat from core client for 30 sec - exiting
09:12:43 (4275): No heartbeat from core client for 30 sec - exiting
09:12:44 (4275): No heartbeat from core client for 30 sec - exiting
09:12:45 (4275): No heartbeat from core client for 30 sec - exiting
09:12:46 (4275): No heartbeat from core client for 30 sec - exiting
09:12:47 (4275): No heartbeat from core client for 30 sec - exiting
09:12:48 (4275): No heartbeat from core client for 30 sec - exiting
09:12:49 (4275): No heartbeat from core client for 30 sec - exiting
09:12:50 (4275): No heartbeat from core client for 30 sec - exiting
09:12:51 (4275): No heartbeat from core client for 30 sec - exiting
09:12:52 (4275): No heartbeat from core client for 30 sec - exiting
09:12:53 (4275): No heartbeat from core client for 30 sec - exiting
09:12:54 (4275): No heartbeat from core client for 30 sec - exiting
09:12:55 (4275): No heartbeat from core client for 30 sec - exiting
09:12:56 (4275): No heartbeat from core client for 30 sec - exiting
09:12:57 (4275): No heartbeat from core client for 30 sec - exiting
09:12:58 (4275): No heartbeat from core client for 30 sec - exiting
09:12:59 (4275): No heartbeat from core client for 30 sec - exiting
09:13:00 (4275): No heartbeat from core client for 30 sec - exiting
10:02:02 (4275): No heartbeat from core client for 30 sec - exiting
10:02:04 (4275): No heartbeat from core client for 30 sec - exiting
10:02:05 (4275): No heartbeat from core client for 30 sec - exiting
10:02:06 (4275): No heartbeat from core client for 30 sec - exiting
10:02:07 (4275): No heartbeat from core client for 30 sec - exiting
10:02:08 (4275): No heartbeat from core client for 30 sec - exiting
10:02:09 (4275): No heartbeat from core client for 30 sec - exiting
10:02:10 (4275): No heartbeat from core client for 30 sec - exiting
10:07:43 (4275): No heartbeat from core client for 30 sec - exiting
10:07:46 (4275): No heartbeat from core client for 30 sec - exiting
10:07:47 (4275): No heartbeat from core client for 30 sec - exiting
10:07:48 (4275): No heartbeat from core client for 30 sec - exiting
10:07:49 (4275): No heartbeat from core client for 30 sec - exiting
01:22:03 (4275): No heartbeat from core client for 30 sec - exiting
01:22:04 (4275): No heartbeat from core client for 30 sec - exiting
10:26:32 (4275): No heartbeat from core client for 30 sec - exiting
10:26:34 (4275): No heartbeat from core client for 30 sec - exiting
10:26:35 (4275): No heartbeat from core client for 30 sec - exiting
10:26:36 (4275): No heartbeat from core client for 30 sec - exiting
10:26:37 (4275): No heartbeat from core client for 30 sec - exiting
10:26:38 (4275): No heartbeat from core client for 30 sec - exiting
10:26:39 (4275): No heartbeat from core client for 30 sec - exiting
10:26:40 (4275): No heartbeat from core client for 30 sec - exiting
10:26:41 (4275): No heartbeat from core client for 30 sec - exiting
08:56:23 (4275): No heartbeat from core client for 30 sec - exiting
08:56:25 (4275): No heartbeat from core client for 30 sec - exiting
08:56:26 (4275): No heartbeat from core client for 30 sec - exiting
08:56:27 (4275): No heartbeat from core client for 30 sec - exiting
08:56:28 (4275): No heartbeat from core client for 30 sec - exiting
08:56:29 (4275): No heartbeat from core client for 30 sec - exiting
08:56:30 (4275): No heartbeat from core client for 30 sec - exiting
08:56:31 (4275): No heartbeat from core client for 30 sec - exiting
08:56:32 (4275): No heartbeat from core client for 30 sec - exiting
08:56:33 (4275): No heartbeat from core client for 30 sec - exiting
22:45:11 (4275): No heartbeat from core client for 30 sec - exiting
22:45:13 (4275): No heartbeat from core client for 30 sec - exiting
22:45:14 (4275): No heartbeat from core client for 30 sec - exiting
22:45:15 (4275): No heartbeat from core client for 30 sec - exiting
22:45:16 (4275): No heartbeat from core client for 30 sec - exiting
09:31:35 (4275): No heartbeat from core client for 30 sec - exiting
09:31:37 (4275): No heartbeat from core client for 30 sec - exiting
09:31:38 (4275): No heartbeat from core client for 30 sec - exiting
09:31:39 (4275): No heartbeat from core client for 30 sec - exiting
09:31:40 (4275): No heartbeat from core client for 30 sec - exiting
09:31:41 (4275): No heartbeat from core client for 30 sec - exiting
09:31:42 (4275): No heartbeat from core client for 30 sec - exiting
09:31:43 (4275): No heartbeat from core client for 30 sec - exiting
09:31:44 (4275): No heartbeat from core client for 30 sec - exiting
09:31:45 (4275): No heartbeat from core client for 30 sec - exiting
09:31:46 (4275): No heartbeat from core client for 30 sec - exiting
09:31:47 (4275): No heartbeat from core client for 30 sec - exiting
01:14:17 (4275): No heartbeat from core client for 30 sec - exiting
01:14:19 (4275): No heartbeat from core client for 30 sec - exiting
08:39:00 (4275): No heartbeat from core client for 30 sec - exiting
08:39:01 (4275): No heartbeat from core client for 30 sec - exiting
08:39:02 (4275): No heartbeat from core client for 30 sec - exiting
08:39:03 (4275): No heartbeat from core client for 30 sec - exiting
08:39:04 (4275): No heartbeat from core client for 30 sec - exiting
08:39:05 (4275): No heartbeat from core client for 30 sec - exiting
08:39:06 (4275): No heartbeat from core client for 30 sec - exiting
09:47:25 (4275): No heartbeat from core client for 30 sec - exiting
04:00:17 (4275): No heartbeat from core client for 30 sec - exiting
04:00:19 (4275): No heartbeat from core client for 30 sec - exiting
04:00:20 (4275): No heartbeat from core client for 30 sec - exiting
15:38:22 (4275): No heartbeat from core client for 30 sec - exiting
15:38:24 (4275): No heartbeat from core client for 30 sec - exiting
15:38:25 (4275): No heartbeat from core client for 30 sec - exiting
15:38:26 (4275): No heartbeat from core client for 30 sec - exiting
15:38:27 (4275): No heartbeat from core client for 30 sec - exiting
15:38:28 (4275): No heartbeat from core client for 30 sec - exiting
15:38:29 (4275): No heartbeat from core client for 30 sec - exiting
15:38:30 (4275): No heartbeat from core client for 30 sec - exiting
15:38:31 (4275): No heartbeat from core client for 30 sec - exiting
15:38:32 (4275): No heartbeat from core client for 30 sec - exiting
10:46:16 (4275): No heartbeat from core client for 30 sec - exiting
10:46:18 (4275): No heartbeat from core client for 30 sec - exiting
10:46:19 (4275): No heartbeat from core client for 30 sec - exiting
10:46:20 (4275): No heartbeat from core client for 30 sec - exiting
10:46:21 (4275): No heartbeat from core client for 30 sec - exiting
10:46:22 (4275): No heartbeat from core client for 30 sec - exiting
10:46:23 (4275): No heartbeat from core client for 30 sec - exiting
10:46:24 (4275): No heartbeat from core client for 30 sec - exiting
10:46:25 (4275): No heartbeat from core client for 30 sec - exiting
10:46:26 (4275): No heartbeat from core client for 30 sec - exiting
10:46:27 (4275): No heartbeat from core client for 30 sec - exiting
09:06:42 (4275): No heartbeat from core client for 30 sec - exiting
09:06:44 (4275): No heartbeat from core client for 30 sec - exiting
09:06:45 (4275): No heartbeat from core client for 30 sec - exiting
09:06:46 (4275): No heartbeat from core client for 30 sec - exiting
09:06:47 (4275): No heartbeat from core client for 30 sec - exiting
09:06:48 (4275): No heartbeat from core client for 30 sec - exiting
01:20:40 (4275): No heartbeat from core client for 30 sec - exiting
01:20:42 (4275): No heartbeat from core client for 30 sec - exiting
01:20:43 (4275): No heartbeat from core client for 30 sec - exiting
01:20:44 (4275): No heartbeat from core client for 30 sec - exiting
01:20:45 (4275): No heartbeat from core client for 30 sec - exiting
01:20:46 (4275): No heartbeat from core client for 30 sec - exiting
09:38:49 (4275): No heartbeat from core client for 30 sec - exiting
09:38:52 (4275): No heartbeat from core client for 30 sec - exiting
09:38:53 (4275): No heartbeat from core client for 30 sec - exiting
09:38:54 (4275): No heartbeat from core client for 30 sec - exiting
09:38:55 (4275): No heartbeat from core client for 30 sec - exiting
09:38:56 (4275): No heartbeat from core client for 30 sec - exiting
09:38:57 (4275): No heartbeat from core client for 30 sec - exiting
09:38:58 (4275): No heartbeat from core client for 30 sec - exiting
09:38:59 (4275): No heartbeat from core client for 30 sec - exiting
09:39:00 (4275): No heartbeat from core client for 30 sec - exiting
09:39:01 (4275): No heartbeat from core client for 30 sec - exiting
09:39:02 (4275): No heartbeat from core client for 30 sec - exiting
09:39:03 (4275): No heartbeat from core client for 30 sec - exiting
09:39:04 (4275): No heartbeat from core client for 30 sec - exiting
09:39:05 (4275): No heartbeat from core client for 30 sec - exiting
09:39:06 (4275): No heartbeat from core client for 30 sec - exiting
09:39:07 (4275): No heartbeat from core client for 30 sec - exiting
09:39:08 (4275): No heartbeat from core client for 30 sec - exiting
09:39:09 (4275): No heartbeat from core client for 30 sec - exiting
09:39:10 (4275): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0902d9f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
20:47:59 (4185): No heartbeat from core client for 30 sec - exiting
20:48:00 (4185): No heartbeat from core client for 30 sec - exiting
20:48:01 (4185): No heartbeat from core client for 30 sec - exiting
20:48:02 (4185): No heartbeat from core client for 30 sec - exiting
20:48:03 (4185): No heartbeat from core client for 30 sec - exiting
20:48:04 (4185): No heartbeat from core client for 30 sec - exiting
20:48:05 (4185): No heartbeat from core client for 30 sec - exiting
20:48:06 (4185): No heartbeat from core client for 30 sec - exiting
20:48:07 (4185): No heartbeat from core client for 30 sec - exiting
20:48:08 (4185): No heartbeat from core client for 30 sec - exiting
20:48:09 (4185): No heartbeat from core client for 30 sec - exiting
20:48:11 (4185): No heartbeat from core client for 30 sec - exiting
20:48:12 (4185): No heartbeat from core client for 30 sec - exiting
20:48:13 (4185): No heartbeat from core client for 30 sec - exiting
20:48:14 (4185): No heartbeat from core client for 30 sec - exiting
20:48:15 (4185): No heartbeat from core client for 30 sec - exiting
20:48:16 (4185): No heartbeat from core client for 30 sec - exiting
20:48:17 (4185): No heartbeat from core client for 30 sec - exiting
20:48:18 (4185): No heartbeat from core client for 30 sec - exiting
20:48:19 (4185): No heartbeat from core client for 30 sec - exiting
20:48:20 (4185): No heartbeat from core client for 30 sec - exiting
20:48:21 (4185): No heartbeat from core client for 30 sec - exiting
20:48:22 (4185): No heartbeat from core client for 30 sec - exiting
20:48:23 (4185): No heartbeat from core client for 30 sec - exiting
20:48:24 (4185): No heartbeat from core client for 30 sec - exiting
20:48:25 (4185): No heartbeat from core client for 30 sec - exiting
20:48:26 (4185): No heartbeat from core client for 30 sec - exiting
20:48:27 (4185): No heartbeat from core client for 30 sec - exiting
20:48:28 (4185): No heartbeat from core client for 30 sec - exiting
20:48:29 (4185): No heartbeat from core client for 30 sec - exiting
20:48:30 (4185): No heartbeat from core client for 30 sec - exiting
20:48:31 (4185): No heartbeat from core client for 30 sec - exiting
20:48:32 (4185): No heartbeat from core client for 30 sec - exiting
20:48:33 (4185): No heartbeat from core client for 30 sec - exiting
20:48:34 (4185): No heartbeat from core client for 30 sec - exiting
20:48:35 (4185): No heartbeat from core client for 30 sec - exiting
20:48:36 (4185): No heartbeat from core client for 30 sec - exiting
20:48:37 (4185): No heartbeat from core client for 30 sec - exiting
20:48:38 (4185): No heartbeat from core client for 30 sec - exiting
20:48:39 (4185): No heartbeat from core client for 30 sec - exiting
20:48:40 (4185): No heartbeat from core client for 30 sec - exiting
20:48:41 (4185): No heartbeat from core client for 30 sec - exiting
20:48:42 (4185): No heartbeat from core client for 30 sec - exiting
20:48:43 (4185): No heartbeat from core client for 30 sec - exiting
20:48:44 (4185): No heartbeat from core client for 30 sec - exiting
20:48:45 (4185): No heartbeat from core client for 30 sec - exiting
20:48:46 (4185): No heartbeat from core client for 30 sec - exiting
20:48:47 (4185): No heartbeat from core client for 30 sec - exiting
20:48:48 (4185): No heartbeat from core client for 30 sec - exiting
20:48:49 (4185): No heartbeat from core client for 30 sec - exiting
20:48:50 (4185): No heartbeat from core client for 30 sec - exiting
20:48:51 (4185): No heartbeat from core client for 30 sec - exiting
23:00:14 (4185): No heartbeat from core client for 30 sec - exiting
23:01:04 (4185): No heartbeat from core client for 30 sec - exiting
23:01:05 (4185): No heartbeat from core client for 30 sec - exiting
23:01:06 (4185): No heartbeat from core client for 30 sec - exiting
23:01:07 (4185): No heartbeat from core client for 30 sec - exiting
23:01:09 (4185): No heartbeat from core client for 30 sec - exiting
23:01:10 (4185): No heartbeat from core client for 30 sec - exiting
23:01:11 (4185): No heartbeat from core client for 30 sec - exiting
23:01:12 (4185): No heartbeat from core client for 30 sec - exiting
23:01:13 (4185): No heartbeat from core client for 30 sec - exiting
23:01:14 (4185): No heartbeat from core client for 30 sec - exiting
23:01:15 (4185): No heartbeat from core client for 30 sec - exiting
23:01:16 (4185): No heartbeat from core client for 30 sec - exiting
23:01:17 (4185): No heartbeat from core client for 30 sec - exiting
23:01:18 (4185): No heartbeat from core client for 30 sec - exiting
23:01:19 (4185): No heartbeat from core client for 30 sec - exiting
23:01:20 (4185): No heartbeat from core client for 30 sec - exiting
23:01:21 (4185): No heartbeat from core client for 30 sec - exiting
23:01:22 (4185): No heartbeat from core client for 30 sec - exiting
23:01:23 (4185): No heartbeat from core client for 30 sec - exiting
23:01:24 (4185): No heartbeat from core client for 30 sec - exiting
23:01:25 (4185): No heartbeat from core client for 30 sec - exiting
23:01:26 (4185): No heartbeat from core client for 30 sec - exiting
23:01:27 (4185): No heartbeat from core client for 30 sec - exiting
23:01:28 (4185): No heartbeat from core client for 30 sec - exiting
23:01:29 (4185): No heartbeat from core client for 30 sec - exiting
23:01:30 (4185): No heartbeat from core client for 30 sec - exiting
23:01:31 (4185): No heartbeat from core client for 30 sec - exiting
23:01:32 (4185): No heartbeat from core client for 30 sec - exiting
23:01:36 (4185): No heartbeat from core client for 30 sec - exiting
23:01:37 (4185): No heartbeat from core client for 30 sec - exiting
23:01:38 (4185): No heartbeat from core client for 30 sec - exiting
23:01:39 (4185): No heartbeat from core client for 30 sec - exiting
23:01:40 (4185): No heartbeat from core client for 30 sec - exiting
23:01:41 (4185): No heartbeat from core client for 30 sec - exiting
23:01:42 (4185): No heartbeat from core client for 30 sec - exiting
23:01:43 (4185): No heartbeat from core client for 30 sec - exiting
23:01:44 (4185): No heartbeat from core client for 30 sec - exiting
23:01:45 (4185): No heartbeat from core client for 30 sec - exiting
23:01:46 (4185): No heartbeat from core client for 30 sec - exiting
23:01:47 (4185): No heartbeat from core client for 30 sec - exiting
23:01:48 (4185): No heartbeat from core client for 30 sec - exiting
23:01:49 (4185): No heartbeat from core client for 30 sec - exiting
23:01:50 (4185): No heartbeat from core client for 30 sec - exiting
23:01:51 (4185): No heartbeat from core client for 30 sec - exiting
23:01:52 (4185): No heartbeat from core client for 30 sec - exiting
23:01:53 (4185): No heartbeat from core client for 30 sec - exiting
23:01:54 (4185): No heartbeat from core client for 30 sec - exiting
23:01:55 (4185): No heartbeat from core client for 30 sec - exiting
23:01:56 (4185): No heartbeat from core client for 30 sec - exiting
23:01:57 (4185): No heartbeat from core client for 30 sec - exiting
23:01:58 (4185): No heartbeat from core client for 30 sec - exiting
23:02:30 (4185): No heartbeat from core client for 30 sec - exiting
23:02:48 (4185): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x091009f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:57:44 (3908): No heartbeat from core client for 30 sec - exiting
23:57:45 (3908): No heartbeat from core client for 30 sec - exiting
23:57:46 (3908): No heartbeat from core client for 30 sec - exiting
23:57:47 (3908): No heartbeat from core client for 30 sec - exiting
23:57:48 (3908): No heartbeat from core client for 30 sec - exiting
23:57:49 (3908): No heartbeat from core client for 30 sec - exiting
23:57:50 (3908): No heartbeat from core client for 30 sec - exiting
23:57:51 (3908): No heartbeat from core client for 30 sec - exiting
23:57:52 (3908): No heartbeat from core client for 30 sec - exiting
23:57:53 (3908): No heartbeat from core client for 30 sec - exiting
23:57:54 (3908): No heartbeat from core client for 30 sec - exiting
23:57:55 (3908): No heartbeat from core client for 30 sec - exiting
23:57:56 (3908): No heartbeat from core client for 30 sec - exiting
23:57:57 (3908): No heartbeat from core client for 30 sec - exiting
07:53:58 (3908): No heartbeat from core client for 30 sec - exiting
07:53:59 (3908): No heartbeat from core client for 30 sec - exiting
07:54:00 (3908): No heartbeat from core client for 30 sec - exiting
07:54:01 (3908): No heartbeat from core client for 30 sec - exiting
07:54:02 (3908): No heartbeat from core client for 30 sec - exiting
07:54:03 (3908): No heartbeat from core client for 30 sec - exiting
07:54:04 (3908): No heartbeat from core client for 30 sec - exiting
07:54:05 (3908): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x099939f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:36:43 (4127): No heartbeat from core client for 30 sec - exiting
23:36:44 (4127): No heartbeat from core client for 30 sec - exiting
21:45:05 (4127): No heartbeat from core client for 30 sec - exiting
21:45:07 (4127): No heartbeat from core client for 30 sec - exiting
21:45:08 (4127): No heartbeat from core client for 30 sec - exiting
21:45:09 (4127): No heartbeat from core client for 30 sec - exiting
21:45:10 (4127): No heartbeat from core client for 30 sec - exiting
21:45:11 (4127): No heartbeat from core client for 30 sec - exiting
21:45:12 (4127): No heartbeat from core client for 30 sec - exiting
21:45:13 (4127): No heartbeat from core client for 30 sec - exiting
21:45:14 (4127): No heartbeat from core client for 30 sec - exiting
21:45:15 (4127): No heartbeat from core client for 30 sec - exiting
21:45:16 (4127): No heartbeat from core client for 30 sec - exiting
21:45:17 (4127): No heartbeat from core client for 30 sec - exiting
21:45:18 (4127): No heartbeat from core client for 30 sec - exiting
21:45:19 (4127): No heartbeat from core client for 30 sec - exiting
21:45:20 (4127): No heartbeat from core client for 30 sec - exiting
21:45:21 (4127): No heartbeat from core client for 30 sec - exiting
21:45:22 (4127): No heartbeat from core client for 30 sec - exiting
21:45:23 (4127): No heartbeat from core client for 30 sec - exiting
21:45:24 (4127): No heartbeat from core client for 30 sec - exiting
21:45:25 (4127): No heartbeat from core client for 30 sec - exiting
21:45:26 (4127): No heartbeat from core client for 30 sec - exiting
21:45:27 (4127): No heartbeat from core client for 30 sec - exiting
21:45:28 (4127): No heartbeat from core client for 30 sec - exiting
21:45:29 (4127): No heartbeat from core client for 30 sec - exiting
21:45:30 (4127): No heartbeat from core client for 30 sec - exiting
21:45:31 (4127): No heartbeat from core client for 30 sec - exiting
21:45:32 (4127): No heartbeat from core client for 30 sec - exiting
21:45:33 (4127): No heartbeat from core client for 30 sec - exiting
21:45:34 (4127): No heartbeat from core client for 30 sec - exiting
21:45:35 (4127): No heartbeat from core client for 30 sec - exiting
21:45:36 (4127): No heartbeat from core client for 30 sec - exiting
21:45:37 (4127): No heartbeat from core client for 30 sec - exiting
21:45:38 (4127): No heartbeat from core client for 30 sec - exiting
21:46:10 (4127): No heartbeat from core client for 30 sec - exiting
21:46:11 (4127): No heartbeat from core client for 30 sec - exiting
21:46:12 (4127): No heartbeat from core client for 30 sec - exiting
21:46:13 (4127): No heartbeat from core client for 30 sec - exiting
21:46:14 (4127): No heartbeat from core client for 30 sec - exiting
21:46:15 (4127): No heartbeat from core client for 30 sec - exiting
21:46:16 (4127): No heartbeat from core client for 30 sec - exiting
21:46:17 (4127): No heartbeat from core client for 30 sec - exiting
21:46:18 (4127): No heartbeat from core client for 30 sec - exiting
21:46:19 (4127): No heartbeat from core client for 30 sec - exiting
21:46:20 (4127): No heartbeat from core client for 30 sec - exiting
21:46:21 (4127): No heartbeat from core client for 30 sec - exiting
21:46:22 (4127): No heartbeat from core client for 30 sec - exiting
21:46:23 (4127): No heartbeat from core client for 30 sec - exiting
21:47:01 (4127): No heartbeat from core client for 30 sec - exiting
21:47:02 (4127): No heartbeat from core client for 30 sec - exiting
21:47:03 (4127): No heartbeat from core client for 30 sec - exiting
21:47:04 (4127): No heartbeat from core client for 30 sec - exiting
21:47:05 (4127): No heartbeat from core client for 30 sec - exiting
21:47:06 (4127): No heartbeat from core client for 30 sec - exiting
21:47:07 (4127): No heartbeat from core client for 30 sec - exiting
21:47:08 (4127): No heartbeat from core client for 30 sec - exiting
21:47:09 (4127): No heartbeat from core client for 30 sec - exiting
21:47:10 (4127): No heartbeat from core client for 30 sec - exiting
21:47:11 (4127): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x097b69f8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
SIGABRT: abort called
22:39:12 (4116): No heartbeat from core client for 30 sec - exiting
22:39:13 (4116): No heartbeat from core client for 30 sec - exiting
22:39:14 (4116): No heartbeat from core client for 30 sec - exiting
22:39:15 (4116): No heartbeat from core client for 30 sec - exiting
22:39:16 (4116): No heartbeat from core client for 30 sec - exiting
22:39:17 (4116): No heartbeat from core client for 30 sec - exiting
22:39:18 (4116): No heartbeat from core client for 30 sec - exiting
22:39:19 (4116): No heartbeat from core client for 30 sec - exiting
22:39:20 (4116): No heartbeat from core client for 30 sec - exiting
22:39:21 (4116): No heartbeat from core client for 30 sec - exiting
22:39:22 (4116): No heartbeat from core client for 30 sec - exiting
22:39:23 (4116): No heartbeat from core client for 30 sec - exiting
22:39:24 (4116): No heartbeat from core client for 30 sec - exiting
22:39:25 (4116): No heartbeat from core client for 30 sec - exiting
22:39:26 (4116): No heartbeat from core client for 30 sec - exiting
09:06:21 (4116): No heartbeat from core client for 30 sec - exiting
09:06:23 (4116): No heartbeat from core client for 30 sec - exiting
09:42:18 (4116): No heartbeat from core client for 30 sec - exiting
09:42:20 (4116): No heartbeat from core client for 30 sec - exiting
09:42:21 (4116): No heartbeat from core client for 30 sec - exiting
09:42:22 (4116): No heartbeat from core client for 30 sec - exiting
09:42:23 (4116): No heartbeat from core client for 30 sec - exiting
09:42:24 (4116): No heartbeat from core client for 30 sec - exiting
01:14:22 (4116): No heartbeat from core client for 30 sec - exiting
01:14:23 (4116): No heartbeat from core client for 30 sec - exiting
01:14:24 (4116): No heartbeat from core client for 30 sec - exiting
01:14:25 (4116): No heartbeat from core client for 30 sec - exiting
01:14:26 (4116): No heartbeat from core client for 30 sec - exiting
01:14:27 (4116): No heartbeat from core client for 30 sec - exiting
01:14:28 (4116): No heartbeat from core client for 30 sec - exiting
01:14:29 (4116): No heartbeat from core client for 30 sec - exiting
01:14:30 (4116): No heartbeat from core client for 30 sec - exiting
01:14:31 (4116): No heartbeat from core client for 30 sec - exiting
01:14:32 (4116): No heartbeat from core client for 30 sec - exiting
01:14:33 (4116): No heartbeat from core client for 30 sec - exiting
01:14:34 (4116): No heartbeat from core client for 30 sec - exiting
01:14:35 (4116): No heartbeat from core client for 30 sec - exiting
01:14:36 (4116): No heartbeat from core client for 30 sec - exiting
01:14:37 (4116): No heartbeat from core client for 30 sec - exiting
01:14:38 (4116): No heartbeat from core client for 30 sec - exiting
01:14:39 (4116): No heartbeat from core client for 30 sec - exiting
01:14:40 (4116): No heartbeat from core client for 30 sec - exiting
01:14:41 (4116): No heartbeat from core client for 30 sec - exiting
01:14:42 (4116): No heartbeat from core client for 30 sec - exiting
01:14:43 (4116): No heartbeat from core client for 30 sec - exiting
10:24:15 (4116): No heartbeat from core client for 30 sec - exiting
10:24:17 (4116): No heartbeat from core client for 30 sec - exiting
10:24:18 (4116): No heartbeat from core client for 30 sec - exiting
10:24:19 (4116): No heartbeat from core client for 30 sec - exiting
10:24:20 (4116): No heartbeat from core client for 30 sec - exiting
10:24:21 (4116): No heartbeat from core client for 30 sec - exiting
10:24:22 (4116): No heartbeat from core client for 30 sec - exiting
10:24:23 (4116): No heartbeat from core client for 30 sec - exiting
10:46:07 (4116): No heartbeat from core client for 30 sec - exiting
10:46:09 (4116): No heartbeat from core client for 30 sec - exiting
10:46:10 (4116): No heartbeat from core client for 30 sec - exiting
12:29:18 (4116): No heartbeat from core client for 30 sec - exiting
12:29:19 (4116): No heartbeat from core client for 30 sec - exiting
12:29:20 (4116): No heartbeat from core client for 30 sec - exiting
12:29:21 (4116): No heartbeat from core client for 30 sec - exiting
12:29:22 (4116): No heartbeat from core client for 30 sec - exiting
12:29:23 (4116): No heartbeat from core client for 30 sec - exiting
12:29:24 (4116): No heartbeat from core client for 30 sec - exiting
12:29:26 (4116): No heartbeat from core client for 30 sec - exiting
12:29:27 (4116): No heartbeat from core client for 30 sec - exiting
12:29:28 (4116): No heartbeat from core client for 30 sec - exiting
12:29:29 (4116): No heartbeat from core client for 30 sec - exiting
12:29:30 (4116): No heartbeat from core client for 30 sec - exiting
12:29:31 (4116): No heartbeat from core client for 30 sec - exiting
02:11:47 (4116): No heartbeat from core client for 30 sec - exiting
02:11:48 (4116): No heartbeat from core client for 30 sec - exiting
02:11:49 (4116): No heartbeat from core client for 30 sec - exiting
02:11:50 (4116): No heartbeat from core client for 30 sec - exiting
02:11:51 (4116): No heartbeat from core client for 30 sec - exiting
02:11:52 (4116): No heartbeat from core client for 30 sec - exiting
02:11:53 (4116): No heartbeat from core client for 30 sec - exiting
02:11:54 (4116): No heartbeat from core client for 30 sec - exiting
02:11:55 (4116): No heartbeat from core client for 30 sec - exiting
02:11:56 (4116): No heartbeat from core client for 30 sec - exiting
02:11:57 (4116): No heartbeat from core client for 30 sec - exiting
02:11:58 (4116): No heartbeat from core client for 30 sec - exiting
02:12:30 (4116): No heartbeat from core client for 30 sec - exiting
02:12:31 (4116): No heartbeat from core client for 30 sec - exiting
02:12:32 (4116): No heartbeat from core client for 30 sec - exiting
02:12:33 (4116): No heartbeat from core client for 30 sec - exiting
02:12:34 (4116): No heartbeat from core client for 30 sec - exiting
08:05:20 (4116): No heartbeat from core client for 30 sec - exiting
08:05:52 (4116): No heartbeat from core client for 30 sec - exiting
08:05:53 (4116): No heartbeat from core client for 30 sec - exiting
08:05:54 (4116): No heartbeat from core client for 30 sec - exiting
08:05:55 (4116): No heartbeat from core client for 30 sec - exiting
08:05:56 (4116): No heartbeat from core client for 30 sec - exiting
08:05:57 (4116): No heartbeat from core client for 30 sec - exiting
08:05:58 (4116): No heartbeat from core client for 30 sec - exiting
12:59:29 (4116): No heartbeat from core client for 30 sec - exiting
12:59:31 (4116): No heartbeat from core client for 30 sec - exiting
12:59:32 (4116): No heartbeat from core client for 30 sec - exiting
12:59:33 (4116): No heartbeat from core client for 30 sec - exiting
13:00:05 (4116): No heartbeat from core client for 30 sec - exiting
13:00:58 (4116): No heartbeat from core client for 30 sec - exiting
13:00:59 (4116): No heartbeat from core client for 30 sec - exiting
13:01:00 (4116): No heartbeat from core client for 30 sec - exiting
13:01:01 (4116): No heartbeat from core client for 30 sec - exiting
13:01:02 (4116): No heartbeat from core client for 30 sec - exiting
13:01:03 (4116): No heartbeat from core client for 30 sec - exiting
13:34:16 (4116): No heartbeat from core client for 30 sec - exiting
13:34:20 (4116): No heartbeat from core client for 30 sec - exiting
13:34:21 (4116): No heartbeat from core client for 30 sec - exiting
13:34:22 (4116): No heartbeat from core client for 30 sec - exiting
13:34:23 (4116): No heartbeat from core client for 30 sec - exiting
13:34:24 (4116): No heartbeat from core client for 30 sec - exiting
13:34:25 (4116): No heartbeat from core client for 30 sec - exiting
13:34:26 (4116): No heartbeat from core client for 30 sec - exiting
13:34:27 (4116): No heartbeat from core client for 30 sec - exiting
13:34:28 (4116): No heartbeat from core client for 30 sec - exiting
13:35:26 (4116): No heartbeat from core client for 30 sec - exiting
16:23:16 (4116): No heartbeat from core client for 30 sec - exiting
16:24:54 (4116): No heartbeat from core client for 30 sec - exiting
04:16:42 (4116): No heartbeat from core client for 30 sec - exiting
04:16:44 (4116): No heartbeat from core client for 30 sec - exiting
04:16:45 (4116): No heartbeat from core client for 30 sec - exiting
09:19:31 (4116): No heartbeat from core client for 30 sec - exiting
09:19:32 (4116): No heartbeat from core client for 30 sec - exiting
09:19:33 (4116): No heartbeat from core client for 30 sec - exiting
09:19:34 (4116): No heartbeat from core client for 30 sec - exiting
23:31:37 (4116): No heartbeat from core client for 30 sec - exiting
23:31:39 (4116): No heartbeat from core client for 30 sec - exiting
23:31:40 (4116): No heartbeat from core client for 30 sec - exiting
23:31:41 (4116): No heartbeat from core client for 30 sec - exiting
23:31:42 (4116): No heartbeat from core client for 30 sec - exiting
23:31:43 (4116): No heartbeat from core client for 30 sec - exiting
23:31:44 (4116): No heartbeat from core client for 30 sec - exiting
23:31:45 (4116): No heartbeat from core client for 30 sec - exiting
18:42:16 (4116): No heartbeat from core client for 30 sec - exiting
18:42:17 (4116): No heartbeat from core client for 30 sec - exiting
18:42:18 (4116): No heartbeat from core client for 30 sec - exiting
18:42:19 (4116): No heartbeat from core client for 30 sec - exiting
18:42:20 (4116): No heartbeat from core client for 30 sec - exiting
18:42:21 (4116): No heartbeat from core client for 30 sec - exiting
18:42:22 (4116): No heartbeat from core client for 30 sec - exiting
18:42:23 (4116): No heartbeat from core client for 30 sec - exiting
18:42:24 (4116): No heartbeat from core client for 30 sec - exiting
18:42:25 (4116): No heartbeat from core client for 30 sec - exiting
18:42:26 (4116): No heartbeat from core client for 30 sec - exiting
18:42:27 (4116): No heartbeat from core client for 30 sec - exiting
18:42:28 (4116): No heartbeat from core client for 30 sec - exiting
18:42:29 (4116): No heartbeat from core client for 30 sec - exiting
18:42:30 (4116): No heartbeat from core client for 30 sec - exiting
23:59:26 (4116): No heartbeat from core client for 30 sec - exiting
23:16:55 (4116): No heartbeat from core client for 30 sec - exiting
23:16:56 (4116): No heartbeat from core client for 30 sec - exiting
23:16:57 (4116): No heartbeat from core client for 30 sec - exiting
23:16:58 (4116): No heartbeat from core client for 30 sec - exiting
23:16:59 (4116): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0875d9f0 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6938b)[0xf73cf38b]
/lib32/libc.so.6(+0x6f440)[0xf73d5440]
/lib32/libc.so.6(+0x700a2)[0xf73d60a2]
/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6(_ZdlPv+0x18)[0xf75c84a8]
/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6(_ZdaPv+0x18)[0xf75c84e8]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80510cc]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xf3)[0xf737e563]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 00:bb 29608415                           /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 00:bb 29608415                           /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
08708000-08774000 rw-p 00000000 00:00 0                                  [heap]
f6d00000-f6d21000 rw-p 00000000 00:00 0 
f6d21000-f6e00000 ---p 00000000 00:00 0 
f6ee6000-f7361000 rw-s 00000000 00:bb 35134491                           /var/lib/boinc/slots/12/139160
f7366000-f74fa000 r-xp 00000000 00:13 29465                              /lib32/libc-2.22.so
f74fa000-f74fb000 ---p 00194000 00:13 29465                              /lib32/libc-2.22.so
f74fb000-f74fd000 r--p 00194000 00:13 29465                              /lib32/libc-2.22.so
f74fd000-f74fe000 rw-p 00196000 00:13 29465                              /lib32/libc-2.22.so
f74fe000-f7501000 rw-p 00000000 00:00 0 
f7506000-f7520000 r-xp 00000000 00:13 6068798                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libgcc_s.so.1
f7520000-f7521000 r--p 00019000 00:13 6068798                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libgcc_s.so.1
f7521000-f7522000 rw-p 0001a000 00:13 6068798                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libgcc_s.so.1
f7526000-f756a000 r-xp 00000000 00:13 29486                              /lib32/libm-2.22.so
f756a000-f756b000 r--p 00043000 00:13 29486                              /lib32/libm-2.22.so
f756b000-f756c000 rw-p 00044000 00:13 29486                              /lib32/libm-2.22.so
f756e000-f769f000 r-xp 00000000 00:13 6068795                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6.0.20
f769f000-f76a4000 r--p 00130000 00:13 6068795                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6.0.20
f76a4000-f76a7000 rw-p 00135000 00:13 6068795                            /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6.0.20
f76a7000-f76ad000 rw-p 00000000 00:00 0 
f76ae000-f76b0000 r-xp 00000000 00:13 29485                              /lib32/libdl-2.22.so
f76b0000-f76b1000 ---p 00002000 00:13 29485                              /lib32/libdl-2.22.so
f76b1000-f76b2000 r--p 00002000 00:13 29485                              /lib32/libdl-2.22.so
f76b2000-f76b3000 rw-p 00003000 00:13 29485                              /lib32/libdl-2.22.so
f76b6000-f76cd000 r-xp 00000000 00:13 29482                              /lib32/libpthread-2.22.so
f76cd000-f76ce000 r--p 00016000 00:13 29482                              /lib32/libpthread-2.22.so
f76ce000-f76cf000 rw-p 00017000 00:13 29482                              /lib32/libpthread-2.22.so
f76cf000-f76d1000 rw-p 00000000 00:00 0 
f771a000-f771b000 ---p 00000000 00:00 0 
f771b000-f771e000 rw-p 00000000 00:00 0                                  [stack:10929]
f771e000-f7720000 rw-s 00000000 00:bb 35134397                           /var/lib/boinc/slots/12/boinc_mmap_file
f7721000-f7726000 rw-p 00000000 00:00 0 
f7726000-f7746000 r-xp 00000000 00:13 29464                              /lib32/ld-2.22.so
f7746000-f7747000 r--p 0001f000 00:13 29464                              /lib32/ld-2.22.so
f7747000-f7748000 rw-p 00020000 00:13 29464                              /lib32/ld-2.22.so
f7748000-f774a000 rw-p 00000000 00:00 0 
f774a000-f774c000 r--p 00000000 00:00 0                                  [vvar]
f774c000-f774e000 r-xp 00000000 00:00 0                                  [vdso]
fff71000-fffe1000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (17 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf774cc40]
linux-gate.so.1(__kernel_vsyscall+0x9)[0xf774cc69]
/lib32/libc.so.6(gsignal+0x47)[0xf7391f07]
/lib32/libc.so.6(abort+0x155)[0xf73935c5]
/lib32/libc.so.6(+0x69390)[0xf73cf390]
/lib32/libc.so.6(+0x6f440)[0xf73d5440]
/lib32/libc.so.6(+0x700a2)[0xf73d60a2]
/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6(_ZdlPv+0x18)[0xf75c84a8]
/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/32/libstdc++.so.6(_ZdaPv+0x18)[0xf75c84e8]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80510cc]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xf3)[0xf737e563]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Feb 2015 07:32:46 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 518,400 709,819 1.3692
13 Feb 2015 11:40:14 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 492,480 674,338 1.3693
12 Feb 2015 03:42:36 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 466,560 639,008 1.3696
11 Feb 2015 10:11:41 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 440,640 603,940 1.3706
10 Feb 2015 18:19:33 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 414,720 568,240 1.3702
09 Feb 2015 11:24:18 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 388,800 533,179 1.3713
08 Feb 2015 18:46:20 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 362,880 498,222 1.3730
08 Feb 2015 04:37:57 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 336,960 463,035 1.3742
06 Feb 2015 19:22:39 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 311,040 427,372 1.3740
06 Feb 2015 16:42:04 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 285,120 391,344 1.3726
03 Feb 2015 08:40:16 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 259,200 355,910 1.3731
01 Feb 2015 05:07:28 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 233,280 320,112 1.3722
30 Jan 2015 07:48:51 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 207,360 284,771 1.3733
28 Jan 2015 17:54:52 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 181,440 249,270 1.3738
26 Jan 2015 00:54:03 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 155,520 213,779 1.3746
23 Jan 2015 02:53:57 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 129,600 178,030 1.3737
21 Jan 2015 12:26:17 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 103,680 142,595 1.3753
20 Jan 2015 22:34:24 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 77,760 106,879 1.3745
20 Jan 2015 08:15:56 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 51,840 71,257 1.3746
19 Jan 2015 18:21:15 1332638 17806711 hadcm3n_xbmu_1940_40_009151372_3 25,920 35,660 1.3758


©2024 climateprediction.net