Task 12986834

Name	hadcm3n_s0o5_1940_40_007298379_0
Workunit	7495803
Created	20 Jun 2011, 12:26:15 UTC
Sent	20 Jun 2011, 12:26:19 UTC
Report deadline	19 Sep 2011, 19:53:30 UTC
Received	4 Jul 2011, 18:21:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1135335
Run time	5 days 15 hours 21 min 33 sec
CPU time	5 days 1 hours 44 min 59 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:04:59 (25447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:00 (25447): No heartbeat from core client for 30 sec - exiting 02:05:01 (25447): No heartbeat from core client for 30 sec - exiting 02:05:02 (25447): No heartbeat from core client for 30 sec - exiting 02:05:03 (25447): No heartbeat from core client for 30 sec - exiting 02:05:04 (25447): No heartbeat from core client for 30 sec - exiting 02:05:05 (25447): No heartbeat from core client for 30 sec - exiting 02:05:06 (25447): No heartbeat from core client for 30 sec - exiting 02:05:07 (25447): No heartbeat from core client for 30 sec - exiting 02:05:08 (25447): No heartbeat from core client for 30 sec - exiting 02:05:09 (25447): No heartbeat from core client for 30 sec - exiting 02:05:10 (25447): No heartbeat from core client for 30 sec - exiting 02:05:11 (25447): No heartbeat from core client for 30 sec - exiting 02:05:12 (25447): No heartbeat from core client for 30 sec - exiting 02:05:13 (25447): No heartbeat from core client for 30 sec - exiting 02:05:14 (25447): No heartbeat from core client for 30 sec - exiting 02:05:15 (25447): No heartbeat from core client for 30 sec - exiting 02:05:16 (25447): No heartbeat from core client for 30 sec - exiting 02:05:17 (25447): No heartbeat from core client for 30 sec - exiting 02:05:18 (25447): No heartbeat from core client for 30 sec - exiting 02:05:19 (25447): No heartbeat from core client for 30 sec - exiting 02:05:20 (25447): No heartbeat from core client for 30 sec - exiting 02:05:21 (25447): No heartbeat from core client for 30 sec - exiting 02:05:22 (25447): No heartbeat from core client for 30 sec - exiting 02:05:23 (25447): No heartbeat from core client for 30 sec - exiting 02:05:24 (25447): No heartbeat from core client for 30 sec - exiting 02:05:25 (25447): No heartbeat from core client for 30 sec - exiting 02:05:26 (25447): No heartbeat from core client for 30 sec - exiting 02:05:27 (25447): No heartbeat from core client for 30 sec - exiting 02:05:28 (25447): No heartbeat from core client for 30 sec - exiting 02:05:29 (25447): No heartbeat from core client for 30 sec - exiting 02:05:30 (25447): No heartbeat from core client for 30 sec - exiting 02:05:31 (25447): No heartbeat from core client for 30 sec - exiting 02:05:32 (25447): No heartbeat from core client for 30 sec - exiting 02:05:33 (25447): No heartbeat from core client for 30 sec - exiting 02:05:34 (25447): No heartbeat from core client for 30 sec - exiting 02:05:35 (25447): No heartbeat from core client for 30 sec - exiting 02:05:36 (25447): No heartbeat from core client for 30 sec - exiting 02:05:37 (25447): No heartbeat from core client for 30 sec - exiting 02:05:38 (25447): No heartbeat from core client for 30 sec - exiting 02:05:39 (25447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:27:53 (7697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/s0o5ko.pje6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ko.pje6c10 Error: Input file: dataout/s0o5ko.pie6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ko.pie6c10 Error: Input file: dataout/s0o5ko.pfe6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ko.pfe6c10 Error: Input file: dataout/s0o5ka.phe6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ka.phe6c10 Error: Input file: dataout/s0o5ka.pge6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ka.pge6c10 Error: Input file: dataout/s0o5ka.pee6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ka.pee6c10 Error: Input file: dataout/s0o5ka.pde6c10 is not a valid UM file. Error converting file to netcdf: dataout/s0o5ka.pde6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2011 17:06:01	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	233,280	439,637	1.8846
03 Jul 2011 02:02:41	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	207,360	389,700	1.8793
01 Jul 2011 07:58:21	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	181,440	340,131	1.8746
30 Jun 2011 04:12:49	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	155,520	290,777	1.8697
28 Jun 2011 21:07:05	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	129,600	242,109	1.8681
27 Jun 2011 04:32:38	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	103,680	194,638	1.8773
23 Jun 2011 04:45:46	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	77,760	146,709	1.8867
22 Jun 2011 06:51:29	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	51,840	97,774	1.8861
21 Jun 2011 08:49:27	1135335	12986834	hadcm3n_s0o5_1940_40_007298379_0	25,920	48,973	1.8894