Task 17655728

Name	hadcm3n_sbsq_1940_40_009111940_4
Workunit	9242276
Created	27 Dec 2014, 2:02:46 UTC
Sent	27 Dec 2014, 2:55:29 UTC
Report deadline	28 Mar 2015, 10:22:40 UTC
Received	14 Mar 2015, 16:58:59 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID	1430660
Run time	6 days 4 hours 24 min 19 sec
CPU time	5 days 21 hours 16 min 42 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.28 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7671, selfPID=7671, iMonCtr=1 hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called Signal 1 received, exiting... Called boinc_finish hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called Signal 1 received, exiting... Called boinc_finish Signal 15 received, exiting... hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called Signal 1 received, exiting... Called boinc_finish hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 13:52:55 (17661): Can't acquire lockfile (-154) - waiting 35s 13:53:30 (17661): Can't acquire lockfile (-154) - exiting 13:54:00 (2372): No heartbeat from core client for 30 sec - exiting 13:54:01 (2372): No heartbeat from core client for 30 sec - exiting 13:54:02 (2372): No heartbeat from core client for 30 sec - exiting 13:54:03 (2372): No heartbeat from core client for 30 sec - exiting 13:54:04 (2372): No heartbeat from core client for 30 sec - exiting 13:54:05 (2372): No heartbeat from core client for 30 sec - exiting 13:54:06 (2372): No heartbeat from core client for 30 sec - exiting 13:54:07 (2372): No heartbeat from core client for 30 sec - exiting 13:54:08 (2372): No heartbeat from core client for 30 sec - exiting 13:54:09 (2372): No heartbeat from core client for 30 sec - exiting 13:54:10 (2372): No heartbeat from core client for 30 sec - exiting 13:54:11 (2372): No heartbeat from core client for 30 sec - exiting 13:54:12 (2372): No heartbeat from core client for 30 sec - exiting 13:54:13 (2372): No heartbeat from core client for 30 sec - exiting 13:54:14 (2372): No heartbeat from core client for 30 sec - exiting 13:54:15 (2372): No heartbeat from core client for 30 sec - exiting 13:54:16 (2372): No heartbeat from core client for 30 sec - exiting 13:54:17 (2372): No heartbeat from core client for 30 sec - exiting 13:54:18 (2372): No heartbeat from core client for 30 sec - exiting 13:54:19 (2372): No heartbeat from core client for 30 sec - exiting 13:54:20 (2372): No heartbeat from core client for 30 sec - exiting 13:54:21 (2372): No heartbeat from core client for 30 sec - exiting 13:54:22 (2372): No heartbeat from core client for 30 sec - exiting 13:54:23 (2372): No heartbeat from core client for 30 sec - exiting 13:54:24 (2372): No heartbeat from core client for 30 sec - exiting 13:54:25 (2372): No heartbeat from core client for 30 sec - exiting 13:54:26 (2372): No heartbeat from core client for 30 sec - exiting 13:54:27 (2372): No heartbeat from core client for 30 sec - exiting 13:54:28 (2372): No heartbeat from core client for 30 sec - exiting 13:54:29 (2372): No heartbeat from core client for 30 sec - exiting 13:54:30 (2372): No heartbeat from core client for 30 sec - exiting 13:54:31 (2372): No heartbeat from core client for 30 sec - exiting 13:54:32 (2372): No heartbeat from core client for 30 sec - exiting 13:54:33 (2372): No heartbeat from core client for 30 sec - exiting 13:54:35 (2372): No heartbeat from core client for 30 sec - exiting 13:54:36 (2372): No heartbeat from core client for 30 sec - exiting 13:54:37 (2372): No heartbeat from core client for 30 sec - exiting 13:54:38 (2372): No heartbeat from core client for 30 sec - exiting 13:54:39 (2372): No heartbeat from core client for 30 sec - exiting 13:54:40 (2372): No heartbeat from core client for 30 sec - exiting 13:54:41 (2372): No heartbeat from core client for 30 sec - exiting 13:54:42 (2372): No heartbeat from core client for 30 sec - exiting 13:54:43 (2372): No heartbeat from core client for 30 sec - exiting 13:54:44 (2372): No heartbeat from core client for 30 sec - exiting 13:54:45 (2372): No heartbeat from core client for 30 sec - exiting 13:54:46 (2372): No heartbeat from core client for 30 sec - exiting 13:54:47 (2372): No heartbeat from core client for 30 sec - exiting 13:54:48 (2372): No heartbeat from core client for 30 sec - exiting 13:54:49 (2372): No heartbeat from core client for 30 sec - exiting 13:54:50 (2372): No heartbeat from core client for 30 sec - exiting 13:54:51 (2372): No heartbeat from core client for 30 sec - exiting 13:54:52 (2372): No heartbeat from core client for 30 sec - exiting 13:54:53 (2372): No heartbeat from core client for 30 sec - exiting 13:54:54 (2372): No heartbeat from core client for 30 sec - exiting 13:54:55 (2372): No heartbeat from core client for 30 sec - exiting 13:54:56 (2372): No heartbeat from core client for 30 sec - exiting 13:54:57 (2372): No heartbeat from core client for 30 sec - exiting 13:54:58 (2372): No heartbeat from core client for 30 sec - exiting 13:54:59 (2372): No heartbeat from core client for 30 sec - exiting 13:55:00 (2372): No heartbeat from core client for 30 sec - exiting 13:55:01 (2372): No heartbeat from core client for 30 sec - exiting 13:55:02 (2372): No heartbeat from core client for 30 sec - exiting 13:55:03 (2372): No heartbeat from core client for 30 sec - exiting 13:55:04 (2372): No heartbeat from core client for 30 sec - exiting 13:55:05 (2372): No heartbeat from core client for 30 sec - exiting 13:55:06 (2372): No heartbeat from core client for 30 sec - exiting 13:55:07 (2372): No heartbeat from core client for 30 sec - exiting 13:55:08 (2372): No heartbeat from core client for 30 sec - exiting 13:55:09 (2372): No heartbeat from core client for 30 sec - exiting 13:55:10 (2372): No heartbeat from core client for 30 sec - exiting 13:55:11 (2372): No heartbeat from core client for 30 sec - exiting 13:55:12 (2372): No heartbeat from core client for 30 sec - exiting 13:55:13 (2372): No heartbeat from core client for 30 sec - exiting 13:55:14 (2372): No heartbeat from core client for 30 sec - exiting 13:55:15 (2372): No heartbeat from core client for 30 sec - exiting 13:55:16 (2372): No heartbeat from core client for 30 sec - exiting 13:55:17 (2372): No heartbeat from core client for 30 sec - exiting 13:55:18 (2372): No heartbeat from core client for 30 sec - exiting 13:55:19 (2372): No heartbeat from core client for 30 sec - exiting 13:55:20 (2372): No heartbeat from core client for 30 sec - exiting 13:55:21 (2372): No heartbeat from core client for 30 sec - exiting 13:55:22 (2372): No heartbeat from core client for 30 sec - exiting 13:55:23 (2372): No heartbeat from core client for 30 sec - exiting 13:55:24 (2372): No heartbeat from core client for 30 sec - exiting 13:55:25 (2372): No heartbeat from core client for 30 sec - exiting 13:55:26 (2372): No heartbeat from core client for 30 sec - exiting 13:55:27 (2372): No heartbeat from core client for 30 sec - exiting 13:55:28 (2372): No heartbeat from core client for 30 sec - exiting 13:55:29 (2372): No heartbeat from core client for 30 sec - exiting 13:55:30 (2372): No heartbeat from core client for 30 sec - exiting 13:55:31 (2372): No heartbeat from core client for 30 sec - exiting 13:55:32 (2372): No heartbeat from core client for 30 sec - exiting 13:55:33 (2372): No heartbeat from core client for 30 sec - exiting 13:55:34 (2372): No heartbeat from core client for 30 sec - exiting 13:55:35 (2372): No heartbeat from core client for 30 sec - exiting 13:55:37 (2372): No heartbeat from core client for 30 sec - exiting 13:55:38 (2372): No heartbeat from core client for 30 sec - exiting 13:55:39 (2372): No heartbeat from core client for 30 sec - exiting 13:55:40 (2372): No heartbeat from core client for 30 sec - exiting 13:55:41 (2372): No heartbeat from core client for 30 sec - exiting 13:55:42 (2372): No heartbeat from core client for 30 sec - exiting 13:55:43 (2372): No heartbeat from core client for 30 sec - exiting 13:55:44 (2372): No heartbeat from core client for 30 sec - exiting 13:55:45 (2372): No heartbeat from core client for 30 sec - exiting 13:55:46 (2372): No heartbeat from core client for 30 sec - exiting 13:55:47 (2372): No heartbeat from core client for 30 sec - exiting 13:55:48 (2372): No heartbeat from core client for 30 sec - exiting 13:55:49 (2372): No heartbeat from core client for 30 sec - exiting 13:55:50 (2372): No heartbeat from core client for 30 sec - exiting 13:55:51 (2372): No heartbeat from core client for 30 sec - exiting 13:55:52 (2372): No heartbeat from core client for 30 sec - exiting 13:55:53 (2372): No heartbeat from core client for 30 sec - exiting 13:55:54 (2372): No heartbeat from core client for 30 sec - exiting 13:55:55 (2372): No heartbeat from core client for 30 sec - exiting 13:55:56 (2372): No heartbeat from core client for 30 sec - exiting 13:55:57 (2372): No heartbeat from core client for 30 sec - exiting 13:55:58 (2372): No heartbeat from core client for 30 sec - exiting 13:55:59 (2372): No heartbeat from core client for 30 sec - exiting 13:56:00 (2372): No heartbeat from core client for 30 sec - exiting 13:56:01 (2372): No heartbeat from core client for 30 sec - exiting 13:56:02 (2372): No heartbeat from core client for 30 sec - exiting 13:56:03 (2372): No heartbeat from core client for 30 sec - exiting 13:56:04 (2372): No heartbeat from core client for 30 sec - exiting 13:56:05 (2372): No heartbeat from core client for 30 sec - exiting 13:56:06 (2372): No heartbeat from core client for 30 sec - exiting 13:56:07 (2372): No heartbeat from core client for 30 sec - exiting 13:56:08 (2372): No heartbeat from core client for 30 sec - exiting 13:56:09 (2372): No heartbeat from core client for 30 sec - exiting 13:56:10 (2372): No heartbeat from core client for 30 sec - exiting 13:56:11 (2372): No heartbeat from core client for 30 sec - exiting 13:56:12 (2372): No heartbeat from core client for 30 sec - exiting 13:56:13 (2372): No heartbeat from core client for 30 sec - exiting 13:56:14 (2372): No heartbeat from core client for 30 sec - exiting 13:56:15 (2372): No heartbeat from core client for 30 sec - exiting 13:56:16 (2372): No heartbeat from core client for 30 sec - exiting 13:56:17 (2372): No heartbeat from core client for 30 sec - exiting 13:56:18 (2372): No heartbeat from core client for 30 sec - exiting 13:56:19 (2372): No heartbeat from core client for 30 sec - exiting 13:56:20 (2372): No heartbeat from core client for 30 sec - exiting 13:56:21 (2372): No heartbeat from core client for 30 sec - exiting 13:56:22 (2372): No heartbeat from core client for 30 sec - exiting 13:56:23 (2372): No heartbeat from core client for 30 sec - exiting 13:56:24 (2372): No heartbeat from core client for 30 sec - exiting 13:56:25 (2372): No heartbeat from core client for 30 sec - exiting 13:56:26 (2372): No heartbeat from core client for 30 sec - exiting 13:56:27 (2372): No heartbeat from core client for 30 sec - exiting 13:56:28 (2372): No heartbeat from core client for 30 sec - exiting 13:56:29 (2372): No heartbeat from core client for 30 sec - exiting 13:56:30 (2372): No heartbeat from core client for 30 sec - exiting 13:56:31 (2372): No heartbeat from core client for 30 sec - exiting 13:56:32 (2372): No heartbeat from core client for 30 sec - exiting 13:56:33 (2372): No heartbeat from core client for 30 sec - exiting 13:56:34 (2372): No heartbeat from core client for 30 sec - exiting 13:56:35 (2372): No heartbeat from core client for 30 sec - exiting 13:56:36 (2372): No heartbeat from core client for 30 sec - exiting 13:56:37 (2372): No heartbeat from core client for 30 sec - exiting 13:56:38 (2372): No heartbeat from core client for 30 sec - exiting 13:56:39 (2372): No heartbeat from core client for 30 sec - exiting 13:56:40 (2372): No heartbeat from core client for 30 sec - exiting 13:56:41 (2372): No heartbeat from core client for 30 sec - exiting 13:56:42 (2372): No heartbeat from core client for 30 sec - exiting 13:56:43 (2372): No heartbeat from core client for 30 sec - exiting 13:56:44 (2372): No heartbeat from core client for 30 sec - exiting 13:56:45 (2372): No heartbeat from core client for 30 sec - exiting 13:56:46 (2372): No heartbeat from core client for 30 sec - exiting 13:56:47 (2372): No heartbeat from core client for 30 sec - exiting 13:56:48 (2372): No heartbeat from core client for 30 sec - exiting 13:56:49 (2372): No heartbeat from core client for 30 sec - exiting 13:56:50 (2372): No heartbeat from core client for 30 sec - exiting 13:56:51 (2372): No heartbeat from core client for 30 sec - exiting 13:56:52 (2372): No heartbeat from core client for 30 sec - exiting 13:56:53 (2372): No heartbeat from core client for 30 sec - exiting 13:56:54 (2372): No heartbeat from core client for 30 sec - exiting 13:56:55 (2372): No heartbeat from core client for 30 sec - exiting 13:56:56 (2372): No heartbeat from core client for 30 sec - exiting 13:56:57 (2372): No heartbeat from core client for 30 sec - exiting 13:56:58 (2372): No heartbeat from core client for 30 sec - exiting 13:56:59 (2372): No heartbeat from core client for 30 sec - exiting 13:57:00 (2372): No heartbeat from core client for 30 sec - exiting 13:57:01 (2372): No heartbeat from core client for 30 sec - exiting 13:57:02 (2372): No heartbeat from core client for 30 sec - exiting 13:57:03 (2372): No heartbeat from core client for 30 sec - exiting 13:57:04 (2372): No heartbeat from core client for 30 sec - exiting 13:57:05 (2372): No heartbeat from core client for 30 sec - exiting 13:57:07 (2372): No heartbeat from core client for 30 sec - exiting 13:57:08 (2372): No heartbeat from core client for 30 sec - exiting 13:57:09 (2372): No heartbeat from core client for 30 sec - exiting 13:57:10 (2372): No heartbeat from core client for 30 sec - exiting 13:57:11 (2372): No heartbeat from core client for 30 sec - exiting 13:57:12 (2372): No heartbeat from core client for 30 sec - exiting 13:57:13 (2372): No heartbeat from core client for 30 sec - exiting 13:57:14 (2372): No heartbeat from core client for 30 sec - exiting 13:57:15 (2372): No heartbeat from core client for 30 sec - exiting 13:57:16 (2372): No heartbeat from core client for 30 sec - exiting 13:57:17 (2372): No heartbeat from core client for 30 sec - exiting 13:57:18 (2372): No heartbeat from core client for 30 sec - exiting 13:57:19 (2372): No heartbeat from core client for 30 sec - exiting 13:57:20 (2372): No heartbeat from core client for 30 sec - exiting 13:57:21 (2372): No heartbeat from core client for 30 sec - exiting 13:57:22 (2372): No heartbeat from core client for 30 sec - exiting 13:57:23 (2372): No heartbeat from core client for 30 sec - exiting 13:57:24 (2372): No heartbeat from core client for 30 sec - exiting 13:57:25 (2372): No heartbeat from core client for 30 sec - exiting 13:57:26 (2372): No heartbeat from core client for 30 sec - exiting 13:57:27 (2372): No heartbeat from core client for 30 sec - exiting 13:57:28 (2372): No heartbeat from core client for 30 sec - exiting 13:57:29 (2372): No heartbeat from core client for 30 sec - exiting 13:57:30 (2372): No heartbeat from core client for 30 sec - exiting 13:57:31 (2372): No heartbeat from core client for 30 sec - exiting 13:57:32 (2372): No heartbeat from core client for 30 sec - exiting 13:57:33 (2372): No heartbeat from core client for 30 sec - exiting 13:57:34 (2372): No heartbeat from core client for 30 sec - exiting 13:57:35 (2372): No heartbeat from core client for 30 sec - exiting 13:57:36 (2372): No heartbeat from core client for 30 sec - exiting 13:57:37 (2372): No heartbeat from core client for 30 sec - exiting 13:57:38 (2372): No heartbeat from core client for 30 sec - exiting 13:57:39 (2372): No heartbeat from core client for 30 sec - exiting 13:57:40 (2372): No heartbeat from core client for 30 sec - exiting 13:57:41 (2372): No heartbeat from core client for 30 sec - exiting 13:57:42 (2372): No heartbeat from core client for 30 sec - exiting 13:57:43 (2372): No heartbeat from core client for 30 sec - exiting 13:57:44 (2372): No heartbeat from core client for 30 sec - exiting 13:57:45 (2372): No heartbeat from core client for 30 sec - exiting 13:57:46 (2372): No heartbeat from core client for 30 sec - exiting 13:57:47 (2372): No heartbeat from core client for 30 sec - exiting 13:57:48 (2372): No heartbeat from core client for 30 sec - exiting 13:57:49 (2372): No heartbeat from core client for 30 sec - exiting 13:57:50 (2372): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called Signal 1 received, exiting... Called boinc_finish hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jan 2015 16:24:04	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	259,200	490,924	1.8940
13 Jan 2015 09:15:39	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	233,280	442,988	1.8990
10 Jan 2015 11:55:30	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	207,360	394,137	1.9007
09 Jan 2015 16:18:10	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	181,440	346,326	1.9088
08 Jan 2015 19:19:42	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	155,520	298,155	1.9171
08 Jan 2015 16:22:06	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	129,600	250,412	1.9322
08 Jan 2015 16:19:08	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	103,680	202,750	1.9555
08 Jan 2015 16:19:08	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	77,760	154,660	1.9889
04 Jan 2015 20:10:18	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	51,840	104,439	2.0146
04 Jan 2015 02:12:10	1330752	17655728	hadcm3n_sbsq_1940_40_009111940_4	25,920	54,054	2.0854