climateprediction.net home page
Task 16150414

Task 16150414

Name hadcm3n_o52a_2060_40_008409980_4
Workunit 8560836
Created 20 Dec 2013, 6:16:51 UTC
Sent 20 Dec 2013, 6:17:04 UTC
Report deadline 21 Mar 2014, 13:44:15 UTC
Received 15 May 2014, 4:31:10 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1292525
Run time 69 days 23 hours 47 min 29 sec
CPU time 12 days 8 hours 7 min 16 sec
Validate state Invalid
Credit 3,421.44
Device peak FLOPS 1.02 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>7.2.7</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
elfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
Model crash detected, will try to restart...
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a0af680 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
04:24:52 (670): No heartbeat from core client for 30 sec - exiting
04:24:53 (670): No heartbeat from core client for 30 sec - exiting
04:24:54 (670): No heartbeat from core client for 30 sec - exiting
04:24:55 (670): No heartbeat from core client for 30 sec - exiting
04:24:56 (670): No heartbeat from core client for 30 sec - exiting
04:24:57 (670): No heartbeat from core client for 30 sec - exiting
04:24:58 (670): No heartbeat from core client for 30 sec - exiting
04:24:59 (670): No heartbeat from core client for 30 sec - exiting
04:25:00 (670): No heartbeat from core client for 30 sec - exiting
04:25:01 (670): No heartbeat from core client for 30 sec - exiting
04:25:02 (670): No heartbeat from core client for 30 sec - exiting
04:25:03 (670): No heartbeat from core client for 30 sec - exiting
04:25:04 (670): No heartbeat from core client for 30 sec - exiting
04:25:05 (670): No heartbeat from core client for 30 sec - exiting
04:25:06 (670): No heartbeat from core client for 30 sec - exiting
04:25:07 (670): No heartbeat from core client for 30 sec - exiting
04:25:08 (670): No heartbeat from core client for 30 sec - exiting
04:25:09 (670): No heartbeat from core client for 30 sec - exiting
04:25:10 (670): No heartbeat from core client for 30 sec - exiting
04:25:11 (670): No heartbeat from core client for 30 sec - exiting
04:25:12 (670): No heartbeat from core client for 30 sec - exiting
04:25:13 (670): No heartbeat from core client for 30 sec - exiting
04:25:14 (670): No heartbeat from core client for 30 sec - exiting
04:25:15 (670): No heartbeat from core client for 30 sec - exiting
04:25:16 (670): No heartbeat from core client for 30 sec - exiting
04:25:17 (670): No heartbeat from core client for 30 sec - exiting
04:25:18 (670): No heartbeat from core client for 30 sec - exiting
04:25:19 (670): No heartbeat from core client for 30 sec - exiting
04:25:20 (670): No heartbeat from core client for 30 sec - exiting
04:25:21 (670): No heartbeat from core client for 30 sec - exiting
04:25:22 (670): No heartbeat from core client for 30 sec - exiting
04:25:23 (670): No heartbeat from core client for 30 sec - exiting
04:25:24 (670): No heartbeat from core client for 30 sec - exiting
04:25:25 (670): No heartbeat from core client for 30 sec - exiting
04:25:26 (670): No heartbeat from core client for 30 sec - exiting
04:25:27 (670): No heartbeat from core client for 30 sec - exiting
04:25:28 (670): No heartbeat from core client for 30 sec - exiting
04:25:29 (670): No heartbeat from core client for 30 sec - exiting
04:25:30 (670): No heartbeat from core client for 30 sec - exiting
04:25:31 (670): No heartbeat from core client for 30 sec - exiting
04:25:32 (670): No heartbeat from core client for 30 sec - exiting
04:25:33 (670): No heartbeat from core client for 30 sec - exiting
04:25:34 (670): No heartbeat from core client for 30 sec - exiting
04:25:35 (670): No heartbeat from core client for 30 sec - exiting
04:25:36 (670): No heartbeat from core client for 30 sec - exiting
04:25:37 (670): No heartbeat from core client for 30 sec - exiting
04:25:38 (670): No heartbeat from core client for 30 sec - exiting
04:25:39 (670): No heartbeat from core client for 30 sec - exiting
04:25:40 (670): No heartbeat from core client for 30 sec - exiting
04:25:41 (670): No heartbeat from core client for 30 sec - exiting
04:25:42 (670): No heartbeat from core client for 30 sec - exiting
04:25:43 (670): No heartbeat from core client for 30 sec - exiting
04:25:44 (670): No heartbeat from core client for 30 sec - exiting
04:25:45 (670): No heartbeat from core client for 30 sec - exiting
04:25:46 (670): No heartbeat from core client for 30 sec - exiting
04:25:47 (670): No heartbeat from core client for 30 sec - exiting
04:25:48 (670): No heartbeat from core client for 30 sec - exiting
04:25:49 (670): No heartbeat from core client for 30 sec - exiting
04:25:50 (670): No heartbeat from core client for 30 sec - exiting
04:25:51 (670): No heartbeat from core client for 30 sec - exiting
04:25:52 (670): No heartbeat from core client for 30 sec - exiting
04:25:53 (670): No heartbeat from core client for 30 sec - exiting
04:25:54 (670): No heartbeat from core client for 30 sec - exiting
04:25:55 (670): No heartbeat from core client for 30 sec - exiting
04:25:56 (670): No heartbeat from core client for 30 sec - exiting
04:25:57 (670): No heartbeat from core client for 30 sec - exiting
04:25:58 (670): No heartbeat from core client for 30 sec - exiting
04:25:59 (670): No heartbeat from core client for 30 sec - exiting
04:26:00 (670): No heartbeat from core client for 30 sec - exiting
04:26:01 (670): No heartbeat from core client for 30 sec - exiting
04:26:02 (670): No heartbeat from core client for 30 sec - exiting
04:26:03 (670): No heartbeat from core client for 30 sec - exiting
04:26:04 (670): No heartbeat from core client for 30 sec - exiting
04:26:05 (670): No heartbeat from core client for 30 sec - exiting
04:26:06 (670): No heartbeat from core client for 30 sec - exiting
04:26:07 (670): No heartbeat from core client for 30 sec - exiting
04:26:08 (670): No heartbeat from core client for 30 sec - exiting
04:26:09 (670): No heartbeat from core client for 30 sec - exiting
04:26:10 (670): No heartbeat from core client for 30 sec - exiting
04:26:11 (670): No heartbeat from core client for 30 sec - exiting
04:26:12 (670): No heartbeat from core client for 30 sec - exiting
04:26:13 (670): No heartbeat from core client for 30 sec - exiting
04:26:14 (670): No heartbeat from core client for 30 sec - exiting
04:26:15 (670): No heartbeat from core client for 30 sec - exiting
04:26:16 (670): No heartbeat from core client for 30 sec - exiting
04:26:17 (670): No heartbeat from core client for 30 sec - exiting
04:26:18 (670): No heartbeat from core client for 30 sec - exiting
04:26:19 (670): No heartbeat from core client for 30 sec - exiting
04:26:20 (670): No heartbeat from core client for 30 sec - exiting
04:26:21 (670): No heartbeat from core client for 30 sec - exiting
04:26:22 (670): No heartbeat from core client for 30 sec - exiting
04:26:23 (670): No heartbeat from core client for 30 sec - exiting
04:26:24 (670): No heartbeat from core client for 30 sec - exiting
04:26:25 (670): No heartbeat from core client for 30 sec - exiting
04:26:26 (670): No heartbeat from core client for 30 sec - exiting
04:26:27 (670): No heartbeat from core client for 30 sec - exiting
04:26:28 (670): No heartbeat from core client for 30 sec - exiting
04:26:29 (670): No heartbeat from core client for 30 sec - exiting
04:26:30 (670): No heartbeat from core client for 30 sec - exiting
04:26:31 (670): No heartbeat from core client for 30 sec - exiting
04:26:32 (670): No heartbeat from core client for 30 sec - exiting
04:26:33 (670): No heartbeat from core client for 30 sec - exiting
04:26:34 (670): No heartbeat from core client for 30 sec - exiting
04:26:35 (670): No heartbeat from core client for 30 sec - exiting
04:26:36 (670): No heartbeat from core client for 30 sec - exiting
04:26:37 (670): No heartbeat from core client for 30 sec - exiting
04:26:38 (670): No heartbeat from core client for 30 sec - exiting
04:26:39 (670): No heartbeat from core client for 30 sec - exiting
04:26:40 (670): No heartbeat from core client for 30 sec - exiting
04:26:41 (670): No heartbeat from core client for 30 sec - exiting
04:26:42 (670): No heartbeat from core client for 30 sec - exiting
04:26:43 (670): No heartbeat from core client for 30 sec - exiting
04:26:44 (670): No heartbeat from core client for 30 sec - exiting
04:26:45 (670): No heartbeat from core client for 30 sec - exiting
04:26:46 (670): No heartbeat from core client for 30 sec - exiting
04:26:47 (670): No heartbeat from core client for 30 sec - exiting
04:26:48 (670): No heartbeat from core client for 30 sec - exiting
04:26:49 (670): No heartbeat from core client for 30 sec - exiting
04:26:50 (670): No heartbeat from core client for 30 sec - exiting
04:26:51 (670): No heartbeat from core client for 30 sec - exiting
04:26:52 (670): No heartbeat from core client for 30 sec - exiting
04:26:53 (670): No heartbeat from core client for 30 sec - exiting
04:26:54 (670): No heartbeat from core client for 30 sec - exiting
04:26:55 (670): No heartbeat from core client for 30 sec - exiting
04:26:56 (670): No heartbeat from core client for 30 sec - exiting
04:26:57 (670): No heartbeat from core client for 30 sec - exiting
04:26:58 (670): No heartbeat from core client for 30 sec - exiting
04:26:59 (670): No heartbeat from core client for 30 sec - exiting
04:27:00 (670): No heartbeat from core client for 30 sec - exiting
04:27:01 (670): No heartbeat from core client for 30 sec - exiting
04:27:02 (670): No heartbeat from core client for 30 sec - exiting
04:27:03 (670): No heartbeat from core client for 30 sec - exiting
04:27:04 (670): No heartbeat from core client for 30 sec - exiting
04:27:05 (670): No heartbeat from core client for 30 sec - exiting
04:27:06 (670): No heartbeat from core client for 30 sec - exiting
04:27:07 (670): No heartbeat from core client for 30 sec - exiting
04:27:08 (670): No heartbeat from core client for 30 sec - exiting
04:27:09 (670): No heartbeat from core client for 30 sec - exiting
04:27:10 (670): No heartbeat from core client for 30 sec - exiting
04:27:11 (670): No heartbeat from core client for 30 sec - exiting
04:27:12 (670): No heartbeat from core client for 30 sec - exiting
04:27:13 (670): No heartbeat from core client for 30 sec - exiting
04:27:14 (670): No heartbeat from core client for 30 sec - exiting
04:27:16 (670): No heartbeat from core client for 30 sec - exiting
04:27:17 (670): No heartbeat from core client for 30 sec - exiting
04:27:18 (670): No heartbeat from core client for 30 sec - exiting
04:27:19 (670): No heartbeat from core client for 30 sec - exiting
04:27:20 (670): No heartbeat from core client for 30 sec - exiting
04:27:21 (670): No heartbeat from core client for 30 sec - exiting
04:27:22 (670): No heartbeat from core client for 30 sec - exiting
04:27:23 (670): No heartbeat from core client for 30 sec - exiting
04:27:24 (670): No heartbeat from core client for 30 sec - exiting
04:27:25 (670): No heartbeat from core client for 30 sec - exiting
04:27:26 (670): No heartbeat from core client for 30 sec - exiting
04:27:27 (670): No heartbeat from core client for 30 sec - exiting
04:27:28 (670): No heartbeat from core client for 30 sec - exiting
04:27:29 (670): No heartbeat from core client for 30 sec - exiting
04:27:30 (670): No heartbeat from core client for 30 sec - exiting
04:27:31 (670): No heartbeat from core client for 30 sec - exiting
04:33:26 (670): No heartbeat from core client for 30 sec - exiting
04:33:27 (670): No heartbeat from core client for 30 sec - exiting
04:33:28 (670): No heartbeat from core client for 30 sec - exiting
04:33:29 (670): No heartbeat from core client for 30 sec - exiting
04:33:30 (670): No heartbeat from core client for 30 sec - exiting
04:33:31 (670): No heartbeat from core client for 30 sec - exiting
04:33:32 (670): No heartbeat from core client for 30 sec - exiting
04:33:33 (670): No heartbeat from core client for 30 sec - exiting
04:33:34 (670): No heartbeat from core client for 30 sec - exiting
04:33:35 (670): No heartbeat from core client for 30 sec - exiting
04:33:36 (670): No heartbeat from core client for 30 sec - exiting
04:33:37 (670): No heartbeat from core client for 30 sec - exiting
04:33:38 (670): No heartbeat from core client for 30 sec - exiting
04:33:39 (670): No heartbeat from core client for 30 sec - exiting
04:33:40 (670): No heartbeat from core client for 30 sec - exiting
04:33:41 (670): No heartbeat from core client for 30 sec - exiting
04:33:42 (670): No heartbeat from core client for 30 sec - exiting
04:33:43 (670): No heartbeat from core client for 30 sec - exiting
04:33:44 (670): No heartbeat from core client for 30 sec - exiting
04:33:45 (670): No heartbeat from core client for 30 sec - exiting
04:33:46 (670): No heartbeat from core client for 30 sec - exiting
04:33:47 (670): No heartbeat from core client for 30 sec - exiting
04:33:48 (670): No heartbeat from core client for 30 sec - exiting
04:33:49 (670): No heartbeat from core client for 30 sec - exiting
04:33:50 (670): No heartbeat from core client for 30 sec - exiting
04:33:51 (670): No heartbeat from core client for 30 sec - exiting
04:33:52 (670): No heartbeat from core client for 30 sec - exiting
04:33:53 (670): No heartbeat from core client for 30 sec - exiting
04:33:54 (670): No heartbeat from core client for 30 sec - exiting
04:33:55 (670): No heartbeat from core client for 30 sec - exiting
04:33:56 (670): No heartbeat from core client for 30 sec - exiting
04:33:57 (670): No heartbeat from core client for 30 sec - exiting
04:33:58 (670): No heartbeat from core client for 30 sec - exiting
04:33:59 (670): No heartbeat from core client for 30 sec - exiting
04:34:00 (670): No heartbeat from core client for 30 sec - exiting
04:34:01 (670): No heartbeat from core client for 30 sec - exiting
04:34:02 (670): No heartbeat from core client for 30 sec - exiting
04:34:03 (670): No heartbeat from core client for 30 sec - exiting
04:34:04 (670): No heartbeat from core client for 30 sec - exiting
04:34:05 (670): No heartbeat from core client for 30 sec - exiting
04:34:06 (670): No heartbeat from core client for 30 sec - exiting
04:34:07 (670): No heartbeat from core client for 30 sec - exiting
04:34:08 (670): No heartbeat from core client for 30 sec - exiting
04:34:09 (670): No heartbeat from core client for 30 sec - exiting
04:34:10 (670): No heartbeat from core client for 30 sec - exiting
04:34:11 (670): No heartbeat from core client for 30 sec - exiting
04:34:12 (670): No heartbeat from core client for 30 sec - exiting
04:34:13 (670): No heartbeat from core client for 30 sec - exiting
04:34:14 (670): No heartbeat from core client for 30 sec - exiting
04:34:15 (670): No heartbeat from core client for 30 sec - exiting
04:34:16 (670): No heartbeat from core client for 30 sec - exiting
04:34:17 (670): No heartbeat from core client for 30 sec - exiting
04:34:18 (670): No heartbeat from core client for 30 sec - exiting
04:34:19 (670): No heartbeat from core client for 30 sec - exiting
04:34:20 (670): No heartbeat from core client for 30 sec - exiting
04:34:21 (670): No heartbeat from core client for 30 sec - exiting
04:34:22 (670): No heartbeat from core client for 30 sec - exiting
04:34:23 (670): No heartbeat from core client for 30 sec - exiting
04:34:24 (670): No heartbeat from core client for 30 sec - exiting
04:34:25 (670): No heartbeat from core client for 30 sec - exiting
04:34:26 (670): No heartbeat from core client for 30 sec - exiting
04:34:27 (670): No heartbeat from core client for 30 sec - exiting
04:34:28 (670): No heartbeat from core client for 30 sec - exiting
04:34:29 (670): No heartbeat from core client for 30 sec - exiting
04:34:30 (670): No heartbeat from core client for 30 sec - exiting
04:34:31 (670): No heartbeat from core client for 30 sec - exiting
04:34:32 (670): No heartbeat from core client for 30 sec - exiting
04:34:33 (670): No heartbeat from core client for 30 sec - exiting
04:34:34 (670): No heartbeat from core client for 30 sec - exiting
04:34:35 (670): No heartbeat from core client for 30 sec - exiting
04:34:36 (670): No heartbeat from core client for 30 sec - exiting
04:34:37 (670): No heartbeat from core client for 30 sec - exiting
04:34:38 (670): No heartbeat from core client for 30 sec - exiting
04:34:39 (670): No heartbeat from core client for 30 sec - exiting
04:34:40 (670): No heartbeat from core client for 30 sec - exiting
04:34:41 (670): No heartbeat from core client for 30 sec - exiting
04:34:42 (670): No heartbeat from core client for 30 sec - exiting
04:34:43 (670): No heartbeat from core client for 30 sec - exiting
04:34:44 (670): No heartbeat from core client for 30 sec - exiting
04:34:45 (670): No heartbeat from core client for 30 sec - exiting
04:34:46 (670): No heartbeat from core client for 30 sec - exiting
04:34:47 (670): No heartbeat from core client for 30 sec - exiting
04:34:48 (670): No heartbeat from core client for 30 sec - exiting
04:34:49 (670): No heartbeat from core client for 30 sec - exiting
04:34:50 (670): No heartbeat from core client for 30 sec - exiting
04:34:51 (670): No heartbeat from core client for 30 sec - exiting
04:34:52 (670): No heartbeat from core client for 30 sec - exiting
04:34:53 (670): No heartbeat from core client for 30 sec - exiting
04:34:54 (670): No heartbeat from core client for 30 sec - exiting
04:34:55 (670): No heartbeat from core client for 30 sec - exiting
04:34:56 (670): No heartbeat from core client for 30 sec - exiting
04:34:57 (670): No heartbeat from core client for 30 sec - exiting
04:34:58 (670): No heartbeat from core client for 30 sec - exiting
04:34:59 (670): No heartbeat from core client for 30 sec - exiting
04:35:00 (670): No heartbeat from core client for 30 sec - exiting
04:35:01 (670): No heartbeat from core client for 30 sec - exiting
04:35:02 (670): No heartbeat from core client for 30 sec - exiting
04:35:03 (670): No heartbeat from core client for 30 sec - exiting
04:35:04 (670): No heartbeat from core client for 30 sec - exiting
04:35:05 (670): No heartbeat from core client for 30 sec - exiting
04:35:06 (670): No heartbeat from core client for 30 sec - exiting
04:35:07 (670): No heartbeat from core client for 30 sec - exiting
04:35:08 (670): No heartbeat from core client for 30 sec - exiting
04:35:09 (670): No heartbeat from core client for 30 sec - exiting
04:35:10 (670): No heartbeat from core client for 30 sec - exiting
04:35:11 (670): No heartbeat from core client for 30 sec - exiting
04:35:12 (670): No heartbeat from core client for 30 sec - exiting
04:35:13 (670): No heartbeat from core client for 30 sec - exiting
04:35:14 (670): No heartbeat from core client for 30 sec - exiting
04:35:15 (670): No heartbeat from core client for 30 sec - exiting
04:35:16 (670): No heartbeat from core client for 30 sec - exiting
04:35:17 (670): No heartbeat from core client for 30 sec - exiting
04:35:18 (670): No heartbeat from core client for 30 sec - exiting
04:35:19 (670): No heartbeat from core client for 30 sec - exiting
04:35:20 (670): No heartbeat from core client for 30 sec - exiting
04:35:21 (670): No heartbeat from core client for 30 sec - exiting
04:35:22 (670): No heartbeat from core client for 30 sec - exiting
04:35:23 (670): No heartbeat from core client for 30 sec - exiting
04:35:24 (670): No heartbeat from core client for 30 sec - exiting
04:35:25 (670): No heartbeat from core client for 30 sec - exiting
04:35:26 (670): No heartbeat from core client for 30 sec - exiting
04:35:27 (670): No heartbeat from core client for 30 sec - exiting
04:35:28 (670): No heartbeat from core client for 30 sec - exiting
04:35:29 (670): No heartbeat from core client for 30 sec - exiting
04:35:30 (670): No heartbeat from core client for 30 sec - exiting
04:35:31 (670): No heartbeat from core client for 30 sec - exiting
04:35:32 (670): No heartbeat from core client for 30 sec - exiting
04:35:33 (670): No heartbeat from core client for 30 sec - exiting
04:35:34 (670): No heartbeat from core client for 30 sec - exiting
04:35:35 (670): No heartbeat from core client for 30 sec - exiting
04:35:36 (670): No heartbeat from core client for 30 sec - exiting
04:35:37 (670): No heartbeat from core client for 30 sec - exiting
04:35:38 (670): No heartbeat from core client for 30 sec - exiting
04:35:39 (670): No heartbeat from core client for 30 sec - exiting
04:35:40 (670): No heartbeat from core client for 30 sec - exiting
04:35:41 (670): No heartbeat from core client for 30 sec - exiting
04:35:42 (670): No heartbeat from core client for 30 sec - exiting
04:35:43 (670): No heartbeat from core client for 30 sec - exiting
04:35:44 (670): No heartbeat from core client for 30 sec - exiting
04:35:45 (670): No heartbeat from core client for 30 sec - exiting
04:35:46 (670): No heartbeat from core client for 30 sec - exiting
04:35:47 (670): No heartbeat from core client for 30 sec - exiting
04:35:48 (670): No heartbeat from core client for 30 sec - exiting
04:35:49 (670): No heartbeat from core client for 30 sec - exiting
04:35:50 (670): No heartbeat from core client for 30 sec - exiting
04:35:51 (670): No heartbeat from core client for 30 sec - exiting
04:35:52 (670): No heartbeat from core client for 30 sec - exiting
04:35:53 (670): No heartbeat from core client for 30 sec - exiting
04:35:54 (670): No heartbeat from core client for 30 sec - exiting
04:35:55 (670): No heartbeat from core client for 30 sec - exiting
04:35:56 (670): No heartbeat from core client for 30 sec - exiting
04:35:57 (670): No heartbeat from core client for 30 sec - exiting
04:35:58 (670): No heartbeat from core client for 30 sec - exiting
04:35:59 (670): No heartbeat from core client for 30 sec - exiting
04:36:00 (670): No heartbeat from core client for 30 sec - exiting
04:36:01 (670): No heartbeat from core client for 30 sec - exiting
04:36:02 (670): No heartbeat from core client for 30 sec - exiting
04:36:03 (670): No heartbeat from core client for 30 sec - exiting
04:43:53 (670): No heartbeat from core client for 30 sec - exiting
04:43:54 (670): No heartbeat from core client for 30 sec - exiting
04:43:55 (670): No heartbeat from core client for 30 sec - exiting
04:43:56 (670): No heartbeat from core client for 30 sec - exiting
04:43:57 (670): No heartbeat from core client for 30 sec - exiting
04:43:58 (670): No heartbeat from core client for 30 sec - exiting
04:43:59 (670): No heartbeat from core client for 30 sec - exiting
04:44:00 (670): No heartbeat from core client for 30 sec - exiting
04:44:01 (670): No heartbeat from core client for 30 sec - exiting
04:44:02 (670): No heartbeat from core client for 30 sec - exiting
04:44:03 (670): No heartbeat from core client for 30 sec - exiting
04:44:04 (670): No heartbeat from core client for 30 sec - exiting
04:44:05 (670): No heartbeat from core client for 30 sec - exiting
04:44:06 (670): No heartbeat from core client for 30 sec - exiting
04:44:07 (670): No heartbeat from core client for 30 sec - exiting
04:44:08 (670): No heartbeat from core client for 30 sec - exiting
04:44:09 (670): No heartbeat from core client for 30 sec - exiting
04:44:10 (670): No heartbeat from core client for 30 sec - exiting
04:44:11 (670): No heartbeat from core client for 30 sec - exiting
04:44:12 (670): No heartbeat from core client for 30 sec - exiting
04:44:13 (670): No heartbeat from core client for 30 sec - exiting
04:44:14 (670): No heartbeat from core client for 30 sec - exiting
04:44:15 (670): No heartbeat from core client for 30 sec - exiting
04:44:16 (670): No heartbeat from core client for 30 sec - exiting
04:44:17 (670): No heartbeat from core client for 30 sec - exiting
04:44:18 (670): No heartbeat from core client for 30 sec - exiting
04:44:19 (670): No heartbeat from core client for 30 sec - exiting
04:44:20 (670): No heartbeat from core client for 30 sec - exiting
04:44:21 (670): No heartbeat from core client for 30 sec - exiting
04:44:22 (670): No heartbeat from core client for 30 sec - exiting
04:44:23 (670): No heartbeat from core client for 30 sec - exiting
04:44:24 (670): No heartbeat from core client for 30 sec - exiting
04:44:25 (670): No heartbeat from core client for 30 sec - exiting
04:44:26 (670): No heartbeat from core client for 30 sec - exiting
04:44:27 (670): No heartbeat from core client for 30 sec - exiting
04:44:28 (670): No heartbeat from core client for 30 sec - exiting
04:44:29 (670): No heartbeat from core client for 30 sec - exiting
04:44:30 (670): No heartbeat from core client for 30 sec - exiting
04:44:31 (670): No heartbeat from core client for 30 sec - exiting
04:44:32 (670): No heartbeat from core client for 30 sec - exiting
04:44:33 (670): No heartbeat from core client for 30 sec - exiting
04:44:34 (670): No heartbeat from core client for 30 sec - exiting
04:44:35 (670): No heartbeat from core client for 30 sec - exiting
04:44:36 (670): No heartbeat from core client for 30 sec - exiting
04:44:37 (670): No heartbeat from core client for 30 sec - exiting
04:44:38 (670): No heartbeat from core client for 30 sec - exiting
04:44:39 (670): No heartbeat from core client for 30 sec - exiting
04:44:40 (670): No heartbeat from core client for 30 sec - exiting
04:44:41 (670): No heartbeat from core client for 30 sec - exiting
04:44:42 (670): No heartbeat from core client for 30 sec - exiting
04:44:43 (670): No heartbeat from core client for 30 sec - exiting
04:44:44 (670): No heartbeat from core client for 30 sec - exiting
04:44:45 (670): No heartbeat from core client for 30 sec - exiting
04:44:46 (670): No heartbeat from core client for 30 sec - exiting
04:44:47 (670): No heartbeat from core client for 30 sec - exiting
04:44:48 (670): No heartbeat from core client for 30 sec - exiting
04:44:49 (670): No heartbeat from core client for 30 sec - exiting
04:44:50 (670): No heartbeat from core client for 30 sec - exiting
04:44:51 (670): No heartbeat from core client for 30 sec - exiting
04:44:52 (670): No heartbeat from core client for 30 sec - exiting
04:44:53 (670): No heartbeat from core client for 30 sec - exiting
04:44:54 (670): No heartbeat from core client for 30 sec - exiting
04:44:55 (670): No heartbeat from core client for 30 sec - exiting
04:44:56 (670): No heartbeat from core client for 30 sec - exiting
04:44:57 (670): No heartbeat from core client for 30 sec - exiting
04:44:58 (670): No heartbeat from core client for 30 sec - exiting
04:44:59 (670): No heartbeat from core client for 30 sec - exiting
04:45:00 (670): No heartbeat from core client for 30 sec - exiting
04:45:01 (670): No heartbeat from core client for 30 sec - exiting
04:45:02 (670): No heartbeat from core client for 30 sec - exiting
04:45:03 (670): No heartbeat from core client for 30 sec - exiting
04:45:04 (670): No heartbeat from core client for 30 sec - exiting
04:45:05 (670): No heartbeat from core client for 30 sec - exiting
04:45:06 (670): No heartbeat from core client for 30 sec - exiting
04:45:07 (670): No heartbeat from core client for 30 sec - exiting
04:45:08 (670): No heartbeat from core client for 30 sec - exiting
04:45:09 (670): No heartbeat from core client for 30 sec - exiting
04:45:10 (670): No heartbeat from core client for 30 sec - exiting
04:45:11 (670): No heartbeat from core client for 30 sec - exiting
04:45:12 (670): No heartbeat from core client for 30 sec - exiting
04:45:13 (670): No heartbeat from core client for 30 sec - exiting
04:45:14 (670): No heartbeat from core client for 30 sec - exiting
04:45:15 (670): No heartbeat from core client for 30 sec - exiting
04:45:16 (670): No heartbeat from core client for 30 sec - exiting
04:45:17 (670): No heartbeat from core client for 30 sec - exiting
04:45:18 (670): No heartbeat from core client for 30 sec - exiting
04:45:19 (670): No heartbeat from core client for 30 sec - exiting
04:45:20 (670): No heartbeat from core client for 30 sec - exiting
04:45:21 (670): No heartbeat from core client for 30 sec - exiting
04:45:22 (670): No heartbeat from core client for 30 sec - exiting
04:45:23 (670): No heartbeat from core client for 30 sec - exiting
04:45:24 (670): No heartbeat from core client for 30 sec - exiting
04:45:25 (670): No heartbeat from core client for 30 sec - exiting
04:45:26 (670): No heartbeat from core client for 30 sec - exiting
04:45:27 (670): No heartbeat from core client for 30 sec - exiting
04:45:28 (670): No heartbeat from core client for 30 sec - exiting
04:45:29 (670): No heartbeat from core client for 30 sec - exiting
04:45:30 (670): No heartbeat from core client for 30 sec - exiting
04:45:31 (670): No heartbeat from core client for 30 sec - exiting
04:45:32 (670): No heartbeat from core client for 30 sec - exiting
04:45:33 (670): No heartbeat from core client for 30 sec - exiting
04:45:34 (670): No heartbeat from core client for 30 sec - exiting
04:45:35 (670): No heartbeat from core client for 30 sec - exiting
04:45:36 (670): No heartbeat from core client for 30 sec - exiting
04:45:37 (670): No heartbeat from core client for 30 sec - exiting
04:45:38 (670): No heartbeat from core client for 30 sec - exiting
04:45:39 (670): No heartbeat from core client for 30 sec - exiting
04:45:40 (670): No heartbeat from core client for 30 sec - exiting
04:45:41 (670): No heartbeat from core client for 30 sec - exiting
04:45:42 (670): No heartbeat from core client for 30 sec - exiting
04:45:43 (670): No heartbeat from core client for 30 sec - exiting
04:45:44 (670): No heartbeat from core client for 30 sec - exiting
04:45:45 (670): No heartbeat from core client for 30 sec - exiting
04:45:46 (670): No heartbeat from core client for 30 sec - exiting
04:45:47 (670): No heartbeat from core client for 30 sec - exiting
04:45:48 (670): No heartbeat from core client for 30 sec - exiting
04:45:49 (670): No heartbeat from core client for 30 sec - exiting
04:45:50 (670): No heartbeat from core client for 30 sec - exiting
04:45:51 (670): No heartbeat from core client for 30 sec - exiting
04:45:52 (670): No heartbeat from core client for 30 sec - exiting
04:45:53 (670): No heartbeat from core client for 30 sec - exiting
04:45:54 (670): No heartbeat from core client for 30 sec - exiting
04:45:55 (670): No heartbeat from core client for 30 sec - exiting
04:45:56 (670): No heartbeat from core client for 30 sec - exiting
04:45:57 (670): No heartbeat from core client for 30 sec - exiting
04:45:58 (670): No heartbeat from core client for 30 sec - exiting
04:45:59 (670): No heartbeat from core client for 30 sec - exiting
04:46:00 (670): No heartbeat from core client for 30 sec - exiting
04:46:01 (670): No heartbeat from core client for 30 sec - exiting
04:46:02 (670): No heartbeat from core client for 30 sec - exiting
04:46:03 (670): No heartbeat from core client for 30 sec - exiting
04:46:04 (670): No heartbeat from core client for 30 sec - exiting
04:46:05 (670): No heartbeat from core client for 30 sec - exiting
04:46:06 (670): No heartbeat from core client for 30 sec - exiting
04:46:07 (670): No heartbeat from core client for 30 sec - exiting
04:46:08 (670): No heartbeat from core client for 30 sec - exiting
04:46:09 (670): No heartbeat from core client for 30 sec - exiting
04:46:10 (670): No heartbeat from core client for 30 sec - exiting
04:46:11 (670): No heartbeat from core client for 30 sec - exiting
04:46:12 (670): No heartbeat from core client for 30 sec - exiting
04:46:13 (670): No heartbeat from core client for 30 sec - exiting
04:46:14 (670): No heartbeat from core client for 30 sec - exiting
04:46:15 (670): No heartbeat from core client for 30 sec - exiting
04:46:16 (670): No heartbeat from core client for 30 sec - exiting
04:46:17 (670): No heartbeat from core client for 30 sec - exiting
04:46:18 (670): No heartbeat from core client for 30 sec - exiting
04:46:19 (670): No heartbeat from core client for 30 sec - exiting
04:46:20 (670): No heartbeat from core client for 30 sec - exiting
04:46:21 (670): No heartbeat from core client for 30 sec - exiting
04:46:22 (670): No heartbeat from core client for 30 sec - exiting
04:46:23 (670): No heartbeat from core client for 30 sec - exiting
04:46:24 (670): No heartbeat from core client for 30 sec - exiting
04:46:25 (670): No heartbeat from core client for 30 sec - exiting
04:46:26 (670): No heartbeat from core client for 30 sec - exiting
04:46:27 (670): No heartbeat from core client for 30 sec - exiting
04:46:28 (670): No heartbeat from core client for 30 sec - exiting
04:46:29 (670): No heartbeat from core client for 30 sec - exiting
04:46:30 (670): No heartbeat from core client for 30 sec - exiting
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09681f78 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08b47f78 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x082c6f78 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08cb1fa8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
*** Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09a90f78 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Jan 2014 22:12:37 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 285,120 1,066,036 3.7389
31 Dec 2013 12:10:35 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 259,200 947,444 3.6553
30 Dec 2013 09:57:15 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 233,280 853,241 3.6576
29 Dec 2013 07:12:02 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 207,360 757,191 3.6516
28 Dec 2013 04:19:31 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 181,440 660,559 3.6406
27 Dec 2013 01:25:06 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 155,520 563,782 3.6251
25 Dec 2013 21:51:06 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 129,600 464,791 3.5864
24 Dec 2013 15:50:46 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 103,680 370,346 3.5720
23 Dec 2013 13:57:33 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 77,760 278,199 3.5777
22 Dec 2013 10:34:54 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 51,840 184,598 3.5609
21 Dec 2013 08:11:14 1292525 16150414 hadcm3n_o52a_2060_40_008409980_4 25,920 91,458 3.5285


©2024 climateprediction.net