Task 15998364

Name	hadcm3n_83rl_1980_40_008462820_1
Workunit	8613676
Created	31 Aug 2013, 18:23:39 UTC
Sent	31 Aug 2013, 18:45:33 UTC
Report deadline	1 Dec 2013, 2:12:44 UTC
Received	16 Sep 2013, 9:35:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153404
Run time	12 days 12 hours 49 min 11 sec
CPU time	10 days 20 hours 48 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.59 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9b8a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19d1404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19eca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19eca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9b8a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x9d4000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19b6c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19b6c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(35647,0xacc7da28) malloc: * error for object 0x19b6c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/3/../../switcher/switcher) is setuid or setgid hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36656,0xacc7da28) malloc: * error for object 0x1246800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_83rl_1980_40_008462820/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Sep 2013 06:20:51	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	777,600	936,451	1.2043
15 Sep 2013 11:24:51	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	751,680	904,906	1.2038
14 Sep 2013 18:30:23	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	725,760	873,272	1.2033
14 Sep 2013 03:35:49	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	699,840	841,619	1.2026
13 Sep 2013 10:46:31	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	673,920	809,837	1.2017
12 Sep 2013 23:13:58	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	648,000	778,052	1.2007
11 Sep 2013 16:48:15	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	622,080	746,766	1.2004
11 Sep 2013 06:15:31	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	596,160	715,704	1.2005
10 Sep 2013 19:12:13	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	570,240	684,567	1.2005
10 Sep 2013 08:54:26	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	544,320	653,474	1.2005
09 Sep 2013 22:29:55	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	518,400	622,382	1.2006
09 Sep 2013 13:15:52	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	492,480	591,353	1.2008
09 Sep 2013 03:51:57	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	466,560	560,283	1.2009
08 Sep 2013 16:59:22	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	440,640	529,103	1.2008
08 Sep 2013 07:16:37	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	414,720	498,046	1.2009
07 Sep 2013 20:57:50	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	388,800	466,761	1.2005
07 Sep 2013 11:34:40	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	362,880	435,663	1.2006
07 Sep 2013 02:17:23	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	336,960	404,596	1.2007
06 Sep 2013 16:49:51	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	311,040	373,482	1.2008
06 Sep 2013 07:21:33	1153404	15998364	hadcm3n_83rl_1980_40_008462820_1	285,120	342,363	1.2008