climateprediction.net home page
Task 16368907

Task 16368907

Name hadcm3n_7ym4_1980_40_008456143_3
Workunit 8606999
Created 14 Mar 2014, 4:52:21 UTC
Sent 14 Mar 2014, 4:52:29 UTC
Report deadline 13 Jun 2014, 12:19:40 UTC
Received 25 Mar 2014, 21:24:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1241465
Run time 11 days 14 hours 3 min 20 sec
CPU time 5 days 7 hours 45 min 58 sec
Validate state Invalid
Credit 10,264.32
Device peak FLOPS 3.50 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.2.42</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  2048    

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  2048    
CPDN Monitor - Quit request from BOINC...
22:17:59 (10913): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:48:56 (12659): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  2048    
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b8400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf75b69d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf770d400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf750b9d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7743400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf75419d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf779e400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf759c9d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77a8400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf75a69d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf777f400]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7]
/home/andrew/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf757d9d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12763, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
25 Mar 2014 15:02:01 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 855,360 724,428 0.8469
25 Mar 2014 07:26:01 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 829,440 697,856 0.8414
24 Mar 2014 23:44:54 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 803,520 671,263 0.8354
24 Mar 2014 16:23:47 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 777,600 644,701 0.8291
24 Mar 2014 08:27:43 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 751,680 618,073 0.8223
24 Mar 2014 00:46:39 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 725,760 591,446 0.8149
23 Mar 2014 18:00:21 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 699,840 564,776 0.8070
23 Mar 2014 09:28:58 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 673,920 538,186 0.7986
23 Mar 2014 01:52:52 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 648,000 511,595 0.7895
22 Mar 2014 18:36:39 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 622,080 484,998 0.7796
22 Mar 2014 10:25:19 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 596,160 569,989 0.9561
22 Mar 2014 02:43:58 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 570,240 543,220 0.9526
21 Mar 2014 19:52:37 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 544,320 516,424 0.9488
21 Mar 2014 11:25:59 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 518,400 489,732 0.9447
21 Mar 2014 04:39:45 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 492,480 462,730 0.9396
20 Mar 2014 18:47:48 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 466,560 432,153 0.9263
20 Mar 2014 09:05:57 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 440,640 400,950 0.9099
19 Mar 2014 23:59:07 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 414,720 369,573 0.8911
19 Mar 2014 14:22:29 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 388,800 336,751 0.8661
19 Mar 2014 05:46:15 1241465 16368907 hadcm3n_7ym4_1980_40_008456143_3 362,880 306,917 0.8458


©2024 climateprediction.net