Task 13130756

Name	hadcm3n_yml8_1900_40_007362134_2
Workunit	7559564
Created	7 Jul 2011, 16:01:10 UTC
Sent	7 Jul 2011, 16:02:18 UTC
Report deadline	6 Oct 2011, 23:29:29 UTC
Received	22 Jul 2011, 21:16:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	994011
Run time	6 days 10 hours 53 min 49 sec
CPU time	6 days 10 hours 53 min 27 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	1.90 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:22:56 (249): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:02 (492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:05 (520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:06 (590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:08 (622): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:08 (678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:40:09 (710): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:09 (728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:09 (736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:10 (797): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:11 (825): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:12 (880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:54:18 (914): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:19 (953): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:58:25 (989): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:26 (1021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:32 (1046): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:33 (1087): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:07:34 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:36 (1188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:41 (1215): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:42 (1249): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:17:43 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:44 (1451): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:45 (1484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:46 (1498): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:46 (1512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:47 (1591): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:48 (1618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:48 (1629): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:49 (1647): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:50 (1705): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:51 (1737): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:51 (1792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:52 (1822): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:52 (1834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:48:52 (1847): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:52:53 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:00 (1947): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:57:01 (1981): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:58:07 (2014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:07 (2048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:08 (2070): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:10 (2141): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:10 (2173): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:10 (2191): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:11 (2206): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:17:12 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:13 (2338): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:23:14 (2402): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:15 (2436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:25:16 (2450): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:27:15 (2461): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:29:16 (2495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:16 (2531): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:17 (2552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yml8ko.pja2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ko.pja2c10 Error: Input file: dataout/yml8ko.pia2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ko.pia2c10 Error: Input file: dataout/yml8ko.pfa2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ko.pfa2c10 Error: Input file: dataout/yml8ka.pha2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ka.pha2c10 Error: Input file: dataout/yml8ka.pga2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ka.pga2c10 Error: Input file: dataout/yml8ka.pea2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ka.pea2c10 Error: Input file: dataout/yml8ka.pda2c10 is not a valid UM file. Error converting file to netcdf: dataout/yml8ka.pda2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:36:11 (375): No heartbeat from core client for 30 sec - exiting 04:36:12 (375): No heartbeat from core client for 30 sec - exiting 04:36:13 (375): No heartbeat from core client for 30 sec - exiting 04:36:14 (375): No heartbeat from core client for 30 sec - exiting 04:36:15 (375): No heartbeat from core client for 30 sec - exiting 04:36:16 (375): No heartbeat from core client for 30 sec - exiting 04:36:17 (375): No heartbeat from core client for 30 sec - exiting 04:36:18 (375): No heartbeat from core client for 30 sec - exiting 04:36:19 (375): No heartbeat from core client for 30 sec - exiting 04:36:20 (375): No heartbeat from core client for 30 sec - exiting 04:36:21 (375): No heartbeat from core client for 30 sec - exiting 04:36:22 (375): No heartbeat from core client for 30 sec - exiting 04:36:23 (375): No heartbeat from core client for 30 sec - exiting 04:36:24 (375): No heartbeat from core client for 30 sec - exiting 04:36:25 (375): No heartbeat from core client for 30 sec - exiting 04:36:26 (375): No heartbeat from core client for 30 sec - exiting 04:36:27 (375): No heartbeat from core client for 30 sec - exiting 04:36:28 (375): No heartbeat from core client for 30 sec - exiting 04:36:29 (375): No heartbeat from core client for 30 sec - exiting 04:36:30 (375): No heartbeat from core client for 30 sec - exiting 04:36:31 (375): No heartbeat from core client for 30 sec - exiting 04:36:32 (375): No heartbeat from core client for 30 sec - exiting 04:36:33 (375): No heartbeat from core client for 30 sec - exiting 04:36:34 (375): No heartbeat from core client for 30 sec - exiting 04:36:35 (375): No heartbeat from core client for 30 sec - exiting 04:36:36 (375): No heartbeat from core client for 30 sec - exiting 04:36:37 (375): No heartbeat from core client for 30 sec - exiting 04:36:38 (375): No heartbeat from core client for 30 sec - exiting 04:36:39 (375): No heartbeat from core client for 30 sec - exiting 04:36:40 (375): No heartbeat from core client for 30 sec - exiting 04:36:41 (375): No heartbeat from core client for 30 sec - exiting 04:36:42 (375): No heartbeat from core client for 30 sec - exiting 04:36:43 (375): No heartbeat from core client for 30 sec - exiting 04:36:44 (375): No heartbeat from core client for 30 sec - exiting 04:36:45 (375): No heartbeat from core client for 30 sec - exiting 04:36:47 (375): No heartbeat from core client for 30 sec - exiting 04:36:48 (375): No heartbeat from core client for 30 sec - exiting 04:36:49 (375): No heartbeat from core client for 30 sec - exiting 04:36:50 (375): No heartbeat from core client for 30 sec - exiting 04:36:51 (375): No heartbeat from core client for 30 sec - exiting 04:36:52 (375): No heartbeat from core client for 30 sec - exiting 04:36:54 (375): No heartbeat from core client for 30 sec - exiting 04:36:57 (375): No heartbeat from core client for 30 sec - exiting 04:37:00 (375): No heartbeat from core client for 30 sec - exiting 04:37:01 (375): No heartbeat from core client for 30 sec - exiting 04:37:03 (375): No heartbeat from core client for 30 sec - exiting 04:37:05 (375): No heartbeat from core client for 30 sec - exiting 04:37:06 (375): No heartbeat from core client for 30 sec - exiting 04:37:08 (375): No heartbeat from core client for 30 sec - exiting 04:37:09 (375): No heartbeat from core client for 30 sec - exiting 04:37:12 (375): No heartbeat from core client for 30 sec - exiting 04:37:13 (375): No heartbeat from core client for 30 sec - exiting 04:37:14 (375): No heartbeat from core client for 30 sec - exiting 04:37:16 (375): No heartbeat from core client for 30 sec - exiting 04:37:17 (375): No heartbeat from core client for 30 sec - exiting 04:37:19 (375): No heartbeat from core client for 30 sec - exiting 04:37:20 (375): No heartbeat from core client for 30 sec - exiting 04:37:22 (375): No heartbeat from core client for 30 sec - exiting 04:37:25 (375): No heartbeat from core client for 30 sec - exiting 04:37:30 (375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135715) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9753, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:54:10	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	207,360	520,948	2.5123
25 Jul 2011 17:30:08	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	181,440	456,222	2.5145
25 Jul 2011 15:58:10	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	155,520	390,754	2.5126
25 Jul 2011 14:18:12	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	129,600	325,882	2.5145
25 Jul 2011 13:36:59	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	103,680	260,838	2.5158
25 Jul 2011 13:36:59	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	77,760	196,138	2.5224
10 Jul 2011 20:28:40	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	51,840	129,888	2.5056
09 Jul 2011 04:53:26	994011	13130756	hadcm3n_yml8_1900_40_007362134_2	25,920	64,259	2.4791