Task 15774864

Name	hadcm3n_4n48_1980_40_008365134_0
Workunit	8515993
Created	10 May 2013, 23:32:53 UTC
Sent	11 May 2013, 0:02:10 UTC
Report deadline	10 Aug 2013, 7:29:21 UTC
Received	12 May 2013, 6:40:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282911
Run time	1 days 2 hours 43 min 23 sec
CPU time	1 days 1 hours 18 min 43 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 01:07:32 (12118): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:33 (12118): No heartbeat from core client for 30 sec - exiting 01:07:34 (12118): No heartbeat from core client for 30 sec - exiting 01:07:35 (12118): No heartbeat from core client for 30 sec - exiting 01:07:36 (12118): No heartbeat from core client for 30 sec - exiting 01:07:37 (12118): No heartbeat from core client for 30 sec - exiting 01:07:38 (12118): No heartbeat from core client for 30 sec - exiting 01:07:39 (12118): No heartbeat from core client for 30 sec - exiting 01:07:40 (12118): No heartbeat from core client for 30 sec - exiting 01:07:41 (12118): No heartbeat from core client for 30 sec - exiting 01:07:42 (12118): No heartbeat from core client for 30 sec - exiting 01:07:43 (12118): No heartbeat from core client for 30 sec - exiting 01:07:44 (12118): No heartbeat from core client for 30 sec - exiting 01:07:45 (12118): No heartbeat from core client for 30 sec - exiting 01:07:46 (12118): No heartbeat from core client for 30 sec - exiting 01:07:47 (12118): No heartbeat from core client for 30 sec - exiting 01:07:48 (12118): No heartbeat from core client for 30 sec - exiting 01:07:49 (12118): No heartbeat from core client for 30 sec - exiting 01:12:08 (12214): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:09 (12214): No heartbeat from core client for 30 sec - exiting 01:12:10 (12214): No heartbeat from core client for 30 sec - exiting 01:12:11 (12214): No heartbeat from core client for 30 sec - exiting 01:12:12 (12214): No heartbeat from core client for 30 sec - exiting 01:12:13 (12214): No heartbeat from core client for 30 sec - exiting 01:12:14 (12214): No heartbeat from core client for 30 sec - exiting 01:12:15 (12214): No heartbeat from core client for 30 sec - exiting 01:12:16 (12214): No heartbeat from core client for 30 sec - exiting 01:12:17 (12214): No heartbeat from core client for 30 sec - exiting 01:12:18 (12214): No heartbeat from core client for 30 sec - exiting 01:12:19 (12214): No heartbeat from core client for 30 sec - exiting 01:12:20 (12214): No heartbeat from core client for 30 sec - exiting 01:12:21 (12214): No heartbeat from core client for 30 sec - exiting 01:12:22 (12214): No heartbeat from core client for 30 sec - exiting 01:12:23 (12214): No heartbeat from core client for 30 sec - exiting 01:16:15 (12316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:16 (12316): No heartbeat from core client for 30 sec - exiting 01:16:17 (12316): No heartbeat from core client for 30 sec - exiting 01:16:18 (12316): No heartbeat from core client for 30 sec - exiting 01:16:19 (12316): No heartbeat from core client for 30 sec - exiting 01:16:20 (12316): No heartbeat from core client for 30 sec - exiting 01:16:21 (12316): No heartbeat from core client for 30 sec - exiting 01:16:22 (12316): No heartbeat from core client for 30 sec - exiting 01:16:23 (12316): No heartbeat from core client for 30 sec - exiting 01:16:24 (12316): No heartbeat from core client for 30 sec - exiting 01:16:25 (12316): No heartbeat from core client for 30 sec - exiting 01:16:26 (12316): No heartbeat from core client for 30 sec - exiting 01:16:27 (12316): No heartbeat from core client for 30 sec - exiting 01:16:28 (12316): No heartbeat from core client for 30 sec - exiting 01:16:29 (12316): No heartbeat from core client for 30 sec - exiting 01:16:30 (12316): No heartbeat from core client for 30 sec - exiting 01:20:14 (12408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:15 (12408): No heartbeat from core client for 30 sec - exiting 01:20:16 (12408): No heartbeat from core client for 30 sec - exiting 01:20:17 (12408): No heartbeat from core client for 30 sec - exiting 01:20:18 (12408): No heartbeat from core client for 30 sec - exiting 01:20:19 (12408): No heartbeat from core client for 30 sec - exiting 01:20:20 (12408): No heartbeat from core client for 30 sec - exiting 01:20:21 (12408): No heartbeat from core client for 30 sec - exiting 01:20:22 (12408): No heartbeat from core client for 30 sec - exiting 01:20:23 (12408): No heartbeat from core client for 30 sec - exiting 03:22:16 (12505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:25 (12635): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:07 (12741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:08 (12741): No heartbeat from core client for 30 sec - exiting 03:34:09 (12741): No heartbeat from core client for 30 sec - exiting 03:34:10 (12741): No heartbeat from core client for 30 sec - exiting 03:34:11 (12741): No heartbeat from core client for 30 sec - exiting 03:34:12 (12741): No heartbeat from core client for 30 sec - exiting 03:34:13 (12741): No heartbeat from core client for 30 sec - exiting 03:34:14 (12741): No heartbeat from core client for 30 sec - exiting 03:34:15 (12741): No heartbeat from core client for 30 sec - exiting 03:34:16 (12741): No heartbeat from core client for 30 sec - exiting 03:34:17 (12741): No heartbeat from core client for 30 sec - exiting 03:34:18 (12741): No heartbeat from core client for 30 sec - exiting 03:41:31 (12885): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:32 (12885): No heartbeat from core client for 30 sec - exiting 03:41:33 (12885): No heartbeat from core client for 30 sec - exiting 03:41:34 (12885): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:45:27 (12970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:38 (12970): No heartbeat from core client for 30 sec - exiting 03:49:26 (13053): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:27 (13137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:28 (13137): No heartbeat from core client for 30 sec - exiting 03:53:29 (13137): No heartbeat from core client for 30 sec - exiting 03:53:30 (13137): No heartbeat from core client for 30 sec - exiting 03:53:31 (13137): No heartbeat from core client for 30 sec - exiting 04:20:44 (13217): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (13217): No heartbeat from core client for 30 sec - exiting 04:28:35 (13348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:33:08 (13464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:44 (13556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:30 (13659): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:31 (13659): No heartbeat from core client for 30 sec - exiting 04:44:32 (13659): No heartbeat from core client for 30 sec - exiting 04:44:33 (13659): No heartbeat from core client for 30 sec - exiting 04:44:34 (13659): No heartbeat from core client for 30 sec - exiting 04:44:35 (13659): No heartbeat from core client for 30 sec - exiting 04:44:36 (13659): No heartbeat from core client for 30 sec - exiting 04:44:37 (13659): No heartbeat from core client for 30 sec - exiting 04:44:38 (13659): No heartbeat from core client for 30 sec - exiting 04:44:39 (13659): No heartbeat from core client for 30 sec - exiting 04:44:40 (13659): No heartbeat from core client for 30 sec - exiting 04:44:41 (13659): No heartbeat from core client for 30 sec - exiting 04:44:42 (13659): No heartbeat from core client for 30 sec - exiting 04:44:43 (13659): No heartbeat from core client for 30 sec - exiting 06:47:10 (13743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:08 (13896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:18 (13980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:19 (13980): No heartbeat from core client for 30 sec - exiting 00:12:20 (13980): No heartbeat from core client for 30 sec - exiting 05:59:35 (14846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:29 (15105): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:30 (15105): No heartbeat from core client for 30 sec - exiting 06:03:31 (15105): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e1400] [0xf76e1425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74fe1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7501825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74e94d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77ac400] [0xf77ac425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c91df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75cc825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75b44d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7795400] [0xf7795425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75b21df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75b5825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf759d4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7794400] [0xf7794425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75b11df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75b4825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf759c4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e6400] [0xf76e6425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75031df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7506825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74ee4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e9400] [0xf76e9425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75061df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7509825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74f14d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15215, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 May 2013 19:54:45	1281428	15774864	hadcm3n_4n48_1980_40_008365134_0	25,920	61,515	2.3733