Task 16232018

Name	hadcm3n_7lc7_1980_40_008438938_1
Workunit	8589794
Created	14 Jan 2014, 7:30:25 UTC
Sent	14 Jan 2014, 7:34:20 UTC
Report deadline	15 Apr 2014, 15:01:31 UTC
Received	25 Feb 2014, 18:49:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1267034
Run time	23 days 7 hours 45 min 3 sec
CPU time	22 days 19 hours 13 min 26 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	0.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.6.41</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb76f5400] [0xb76f5416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb751a1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb751d825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75054d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7784400] [0xb7784416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75a91df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75ac825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75944d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77d7400] [0xb77d7416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75fc1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75ff825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75e74d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb76e9400] [0xb76e9416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb750e1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7511825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb74f94d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7732400] [0xb7732416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75571df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb755a825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75424d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb773a400] [0xb773a416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb755f1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7562825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb754a4d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12408, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Feb 2014 10:15:59	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	414,720	1,867,503	4.5030
22 Feb 2014 01:41:37	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	388,800	1,753,320	4.5096
20 Feb 2014 16:39:13	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	362,880	1,638,231	4.5145
19 Feb 2014 08:07:35	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	336,960	1,524,230	4.5235
18 Feb 2014 12:02:43	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	311,040	1,407,371	4.5247
16 Feb 2014 11:54:04	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	285,120	1,290,898	4.5276
15 Feb 2014 02:50:56	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	259,200	1,173,829	4.5287
13 Feb 2014 16:52:02	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	233,280	1,057,028	4.5312
11 Feb 2014 12:01:30	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	207,360	940,626	4.5362
10 Feb 2014 03:41:17	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	181,440	825,973	4.5523
08 Feb 2014 16:50:46	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	155,520	712,435	4.5810
07 Feb 2014 08:00:06	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	129,600	599,074	4.6225
03 Feb 2014 22:32:11	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	103,680	483,801	4.6663
01 Feb 2014 18:20:42	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	77,760	367,603	4.7274
30 Jan 2014 22:50:29	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	51,840	248,009	4.7841
18 Jan 2014 10:22:22	1267034	16232018	hadcm3n_7lc7_1980_40_008438938_1	25,920	118,699	4.5794