climateprediction.net home page
Task 15802490

Task 15802490

Name hadcm3n_n36h_1880_40_008373969_0
Workunit 8524828
Created 29 May 2013, 20:23:31 UTC
Sent 31 May 2013, 18:27:31 UTC
Report deadline 31 Aug 2013, 1:54:42 UTC
Received 23 Jun 2013, 5:04:21 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1240735
Run time 17 days 18 hours 26 min 7 sec
CPU time 15 days 21 hours 31 min 59 sec
Validate state Invalid
Credit 12,441.60
Device peak FLOPS 3.09 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
21:32:24 (4428): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:35:43 (27030): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:35:44 (27030): No heartbeat from core client for 30 sec - exiting
21:35:45 (27030): No heartbeat from core client for 30 sec - exiting
21:35:46 (27030): No heartbeat from core client for 30 sec - exiting
21:35:47 (27030): No heartbeat from core client for 30 sec - exiting
21:35:48 (27030): No heartbeat from core client for 30 sec - exiting
21:35:49 (27030): No heartbeat from core client for 30 sec - exiting
21:35:50 (27030): No heartbeat from core client for 30 sec - exiting
21:35:51 (27030): No heartbeat from core client for 30 sec - exiting
21:35:52 (27030): No heartbeat from core client for 30 sec - exiting
21:35:53 (27030): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Error: Input file: dataout/n36hko.pj81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hko.pj81c10
Error: Input file: dataout/n36hko.pi81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hko.pi81c10
Error: Input file: dataout/n36hko.pf81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hko.pf81c10
Error: Input file: dataout/n36hka.ph81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hka.ph81c10
Error: Input file: dataout/n36hka.pg81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hka.pg81c10
Error: Input file: dataout/n36hka.pe81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hka.pe81c10
Error: Input file: dataout/n36hka.pd81c10 is not a valid UM file.
Error converting file to netcdf: dataout/n36hka.pd81c10
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
10:15:19 (2346): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
10:30:09 (27165): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:30:10 (27165): No heartbeat from core client for 30 sec - exiting
10:30:11 (27165): No heartbeat from core client for 30 sec - exiting
10:34:27 (32720): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
09:24:59 (1718): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:29:19 (20120): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:29:20 (20120): No heartbeat from core client for 30 sec - exiting
12:18:36 (21216): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:21:19 (1850): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:22:03 (2707): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
12:29:25 (3055): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:29:26 (3055): No heartbeat from core client for 30 sec - exiting
12:29:27 (3055): No heartbeat from core client for 30 sec - exiting
12:29:28 (3055): No heartbeat from core client for 30 sec - exiting
12:29:29 (3055): No heartbeat from core client for 30 sec - exiting
12:29:30 (3055): No heartbeat from core client for 30 sec - exiting
12:29:31 (3055): No heartbeat from core client for 30 sec - exiting
12:29:32 (3055): No heartbeat from core client for 30 sec - exiting
12:35:17 (6150): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:35:18 (6150): No heartbeat from core client for 30 sec - exiting
12:35:19 (6150): No heartbeat from core client for 30 sec - exiting
12:35:20 (6150): No heartbeat from core client for 30 sec - exiting
12:35:21 (6150): No heartbeat from core client for 30 sec - exiting
12:35:22 (6150): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
12:46:45 (8368): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:46:46 (8368): No heartbeat from core client for 30 sec - exiting
12:46:47 (8368): No heartbeat from core client for 30 sec - exiting
12:48:49 (10864): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
13:08:06 (11242): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:08:07 (11242): No heartbeat from core client for 30 sec - exiting
13:08:08 (11242): No heartbeat from core client for 30 sec - exiting
13:09:40 (15457): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:09:41 (15457): No heartbeat from core client for 30 sec - exiting
13:09:42 (15457): No heartbeat from core client for 30 sec - exiting
13:09:43 (15457): No heartbeat from core client for 30 sec - exiting
13:09:44 (15457): No heartbeat from core client for 30 sec - exiting
13:09:45 (15457): No heartbeat from core client for 30 sec - exiting
13:09:46 (15457): No heartbeat from core client for 30 sec - exiting
13:09:47 (15457): No heartbeat from core client for 30 sec - exiting
13:09:48 (15457): No heartbeat from core client for 30 sec - exiting
13:09:49 (15457): No heartbeat from core client for 30 sec - exiting
13:09:50 (15457): No heartbeat from core client for 30 sec - exiting
13:09:51 (15457): No heartbeat from core client for 30 sec - exiting
13:09:52 (15457): No heartbeat from core client for 30 sec - exiting
13:09:53 (15457): No heartbeat from core client for 30 sec - exiting
13:09:54 (15457): No heartbeat from core client for 30 sec - exiting
13:09:55 (15457): No heartbeat from core client for 30 sec - exiting
13:09:56 (15457): No heartbeat from core client for 30 sec - exiting
13:09:57 (15457): No heartbeat from core client for 30 sec - exiting
13:09:58 (15457): No heartbeat from core client for 30 sec - exiting
13:09:59 (15457): No heartbeat from core client for 30 sec - exiting
13:10:00 (15457): No heartbeat from core client for 30 sec - exiting
13:10:01 (15457): No heartbeat from core client for 30 sec - exiting
14:30:56 (15869): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:30:57 (15869): No heartbeat from core client for 30 sec - exiting
14:30:58 (15869): No heartbeat from core client for 30 sec - exiting
14:30:59 (15869): No heartbeat from core client for 30 sec - exiting
14:31:00 (15869): No heartbeat from core client for 30 sec - exiting
14:31:01 (15869): No heartbeat from core client for 30 sec - exiting
14:31:02 (15869): No heartbeat from core client for 30 sec - exiting
14:31:03 (15869): No heartbeat from core client for 30 sec - exiting
14:31:04 (15869): No heartbeat from core client for 30 sec - exiting
14:31:05 (15869): No heartbeat from core client for 30 sec - exiting
14:34:33 (29317): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:34:34 (29317): No heartbeat from core client for 30 sec - exiting
14:34:35 (29317): No heartbeat from core client for 30 sec - exiting
14:34:36 (29317): No heartbeat from core client for 30 sec - exiting
14:34:37 (29317): No heartbeat from core client for 30 sec - exiting
15:46:09 (30350): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:46:10 (30350): No heartbeat from core client for 30 sec - exiting
15:46:11 (30350): No heartbeat from core client for 30 sec - exiting
15:46:12 (30350): No heartbeat from core client for 30 sec - exiting
15:46:13 (30350): No heartbeat from core client for 30 sec - exiting
15:46:14 (30350): No heartbeat from core client for 30 sec - exiting
15:46:15 (30350): No heartbeat from core client for 30 sec - exiting
15:46:16 (30350): No heartbeat from core client for 30 sec - exiting
15:46:17 (30350): No heartbeat from core client for 30 sec - exiting
15:46:18 (30350): No heartbeat from core client for 30 sec - exiting
15:46:19 (30350): No heartbeat from core client for 30 sec - exiting
15:46:20 (30350): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
15:50:42 (4151): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:50:43 (4151): No heartbeat from core client for 30 sec - exiting
15:50:44 (4151): No heartbeat from core client for 30 sec - exiting
15:50:45 (4151): No heartbeat from core client for 30 sec - exiting
16:04:23 (5047): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:04:24 (5047): No heartbeat from core client for 30 sec - exiting
16:04:25 (5047): No heartbeat from core client for 30 sec - exiting
16:04:26 (5047): No heartbeat from core client for 30 sec - exiting
16:08:01 (9080): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:08:02 (9080): No heartbeat from core client for 30 sec - exiting
16:08:03 (9080): No heartbeat from core client for 30 sec - exiting

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITHEAD: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITHEAD: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITHEAD: I/O error                                                                                                                            14:07:57 (10085): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:08:01 (10085): No heartbeat from core client for 30 sec - exiting
14:08:02 (10085): No heartbeat from core client for 30 sec - exiting
14:08:03 (10085): No heartbeat from core client for 30 sec - exiting
14:08:04 (10085): No heartbeat from core client for 30 sec - exiting
14:08:05 (10085): No heartbeat from core client for 30 sec - exiting
14:08:06 (10085): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x087aaae0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0910b020 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0871c020 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x095cf040 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0978a040 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x082ce040 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08bee040 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08935040 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
21 Jun 2013 12:20:00 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 1,036,800 1,372,988 1.3243
21 Jun 2013 03:00:50 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 1,010,880 1,340,336 1.3259
20 Jun 2013 17:08:31 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 984,960 1,306,183 1.3261
20 Jun 2013 07:47:32 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 959,040 1,272,712 1.3271
19 Jun 2013 21:06:30 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 933,120 1,341,300 1.4374
19 Jun 2013 10:22:10 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 907,200 1,304,615 1.4381
18 Jun 2013 23:21:41 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 881,280 1,266,681 1.4373
18 Jun 2013 11:52:07 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 855,360 1,230,072 1.4381
18 Jun 2013 00:59:59 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 829,440 1,192,014 1.4371
17 Jun 2013 14:23:12 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 803,520 1,154,151 1.4364
17 Jun 2013 04:44:17 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 777,600 1,116,828 1.4363
16 Jun 2013 18:02:23 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 751,680 1,079,382 1.4360
16 Jun 2013 07:10:04 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 725,760 1,041,839 1.4355
15 Jun 2013 20:34:33 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 699,840 1,004,984 1.4360
15 Jun 2013 09:45:02 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 673,920 967,968 1.4363
14 Jun 2013 23:14:59 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 648,000 930,745 1.4363
14 Jun 2013 12:44:59 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 622,080 893,489 1.4363
14 Jun 2013 02:14:26 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 596,160 856,006 1.4359
13 Jun 2013 15:36:54 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 570,240 818,357 1.4351
13 Jun 2013 05:17:31 1240735 15802490 hadcm3n_n36h_1880_40_008373969_0 544,320 781,326 1.4354


©2024 climateprediction.net