Task 13028473

Name	hadcm3n_t7o2_1940_40_007317252_1
Workunit	7514682
Created	29 Jun 2011, 7:06:47 UTC
Sent	29 Jun 2011, 10:22:48 UTC
Report deadline	28 Sep 2011, 17:49:59 UTC
Received	22 Jul 2011, 14:05:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	817811
Run time	4 days 21 hours 28 min 39 sec
CPU time	3 days 5 hours 21 min 45 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 20:46:34 (2072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:35 (2072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3248, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:29:51 (2688): No heartbeat from core client for 30 sec - exiting 12:29:52 (2688): No heartbeat from core client for 30 sec - exiting 12:29:53 (2688): No heartbeat from core client for 30 sec - exiting 12:29:54 (2688): No heartbeat from core client for 30 sec - exiting 12:29:55 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... 18:11:39 (5956): No heartbeat from core client for 30 sec - exiting 18:11:40 (5956): No heartbeat from core client for 30 sec - exiting 18:11:42 (5956): No heartbeat from core client for 30 sec - exiting 18:11:43 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:42:51 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:04:27 (5908): Can't set up shared mem: -1. Will run in standalone mode. 12:04:56 (3500): Can't set up shared mem: -1. Will run in standalone mode. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5376, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3428, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5424, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4484, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t7o2ko.pje4c10 Error converting file to netcdf: dataout/t7o2ko.pie4c10 Error converting file to netcdf: dataout/t7o2ko.pfe4c10 Error converting file to netcdf: dataout/t7o2ka.phe4c10 Error converting file to netcdf: dataout/t7o2ka.pge4c10 Error converting file to netcdf: dataout/t7o2ka.pee4c10 Error converting file to netcdf: dataout/t7o2ka.pde4c10 21:43:31 (1508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:32 (1508): No heartbeat from core client for 30 sec - exiting 21:43:33 (1508): No heartbeat from core client for 30 sec - exiting 21:43:34 (1508): No heartbeat from core client for 30 sec - exiting 21:43:35 (1508): No heartbeat from core client for 30 sec - exiting 21:43:36 (1508): No heartbeat from core client for 30 sec - exiting 22:46:47 (5040): No heartbeat from core client for 30 sec - exiting 22:46:48 (5040): No heartbeat from core client for 30 sec - exiting 22:46:49 (5040): No heartbeat from core client for 30 sec - exiting 22:46:50 (5040): No heartbeat from core client for 30 sec - exiting 22:46:51 (5040): No heartbeat from core client for 30 sec - exiting 22:46:52 (5040): No heartbeat from core client for 30 sec - exiting 22:46:53 (5040): No heartbeat from core client for 30 sec - exiting 22:46:54 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:55 (5040): No heartbeat from core client for 30 sec - exiting 22:46:56 (5040): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=700, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 18:11:23	817811	13028473	hadcm3n_t7o2_1940_40_007317252_1	103,680	272,569	2.6289
25 Jul 2011 15:48:52	817811	13028473	hadcm3n_t7o2_1940_40_007317252_1	77,760	199,351	2.5637
10 Jul 2011 15:19:23	817811	13028473	hadcm3n_t7o2_1940_40_007317252_1	51,840	138,580	2.6732
02 Jul 2011 15:24:53	817811	13028473	hadcm3n_t7o2_1940_40_007317252_1	25,920	67,763	2.6143