Task 15525849

Name	hadcm3n_o2cu_2140_40_008270392_3
Workunit	8425516
Created	8 Jan 2013, 0:53:27 UTC
Sent	8 Jan 2013, 0:53:34 UTC
Report deadline	9 Apr 2013, 8:20:45 UTC
Received	6 Mar 2013, 22:50:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1064622
Run time	14 days 21 hours 28 min 1 sec
CPU time	14 days 21 hours 28 min 1 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:05:33 (156): No heartbeat from core client for 30 sec - exiting 21:05:34 (156): No heartbeat from core client for 30 sec - exiting 21:05:35 (156): No heartbeat from core client for 30 sec - exiting 21:05:36 (156): No heartbeat from core client for 30 sec - exiting 21:05:37 (156): No heartbeat from core client for 30 sec - exiting 21:05:38 (156): No heartbeat from core client for 30 sec - exiting 21:05:39 (156): No heartbeat from core client for 30 sec - exiting 21:05:40 (156): No heartbeat from core client for 30 sec - exiting 21:05:41 (156): No heartbeat from core client for 30 sec - exiting 21:05:42 (156): No heartbeat from core client for 30 sec - exiting 21:05:43 (156): No heartbeat from core client for 30 sec - exiting 21:05:44 (156): No heartbeat from core client for 30 sec - exiting 21:05:45 (156): No heartbeat from core client for 30 sec - exiting 21:05:46 (156): No heartbeat from core client for 30 sec - exiting 21:05:47 (156): No heartbeat from core client for 30 sec - exiting 21:05:48 (156): No heartbeat from core client for 30 sec - exiting 21:05:49 (156): No heartbeat from core client for 30 sec - exiting 21:05:50 (156): No heartbeat from core client for 30 sec - exiting 21:05:51 (156): No heartbeat from core client for 30 sec - exiting 21:05:52 (156): No heartbeat from core client for 30 sec - exiting 21:05:53 (156): No heartbeat from core client for 30 sec - exiting 21:05:54 (156): No heartbeat from core client for 30 sec - exiting 21:05:55 (156): No heartbeat from core client for 30 sec - exiting 21:05:56 (156): No heartbeat from core client for 30 sec - exiting 21:05:57 (156): No heartbeat from core client for 30 sec - exiting 21:05:58 (156): No heartbeat from core client for 30 sec - exiting 21:05:59 (156): No heartbeat from core client for 30 sec - exiting 21:06:00 (156): No heartbeat from core client for 30 sec - exiting 21:06:01 (156): No heartbeat from core client for 30 sec - exiting 21:06:02 (156): No heartbeat from core client for 30 sec - exiting 21:06:03 (156): No heartbeat from core client for 30 sec - exiting 21:06:04 (156): No heartbeat from core client for 30 sec - exiting 21:06:05 (156): No heartbeat from core client for 30 sec - exiting 21:06:06 (156): No heartbeat from core client for 30 sec - exiting 21:06:07 (156): No heartbeat from core client for 30 sec - exiting 21:06:08 (156): No heartbeat from core client for 30 sec - exiting 21:06:09 (156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:00 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:37 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:56:32 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 01:46:27 AM No files match the supplied pattern. MainError: 01:46:27 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 04:52:52 PM No files match the supplied pattern. MainError: 04:52:52 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:43:36 PM No files match the supplied pattern. MainError: 10:43:36 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 05:47:41 AM No files match the supplied pattern. MainError: 05:47:41 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 08:53:11 PM No files match the supplied pattern. MainError: 08:53:11 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 11:32:01 PM No files match the supplied pattern. MainError: 11:32:01 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 01:00:44 AM No files match the supplied pattern. MainError: 01:00:44 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:29:07 PM No files match the supplied pattern. MainError: 03:29:07 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 05:18:49 PM No files match the supplied pattern. MainError: 05:18:49 PM No files match the supplied pattern. MainError: 05:42:16 PM No files match the supplied pattern. MainError: 05:42:16 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/o2cuka.ph11c10 Error converting file to netcdf: dataout/o2cuka.pg11c10 Error converting file to netcdf: dataout/o2cuka.pe11c10 MainError: 09:49:32 PM No files match the supplied pattern. MainError: 09:49:32 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Mar 2013 21:50:34	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	777,600	1,296,400	1.6672
05 Mar 2013 17:48:59	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	751,680	1,251,102	1.6644
04 Mar 2013 17:34:12	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	725,760	1,207,667	1.6640
02 Mar 2013 15:55:48	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	699,840	1,164,480	1.6639
01 Mar 2013 01:05:36	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	673,920	1,121,959	1.6648
26 Feb 2013 23:34:33	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	648,000	1,078,952	1.6650
24 Feb 2013 20:57:54	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	622,080	1,034,764	1.6634
24 Feb 2013 05:49:12	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	596,160	991,507	1.6632
22 Feb 2013 22:44:50	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	570,240	947,524	1.6616
21 Feb 2013 16:56:00	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	544,320	903,767	1.6604
20 Feb 2013 01:46:41	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	518,400	860,432	1.6598
16 Feb 2013 02:07:34	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	492,480	816,866	1.6587
14 Feb 2013 20:53:36	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	466,560	773,193	1.6572
12 Feb 2013 03:28:27	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	440,640	729,245	1.6550
10 Feb 2013 01:00:06	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	414,720	685,316	1.6525
07 Feb 2013 03:51:16	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	388,800	641,580	1.6502
02 Feb 2013 07:00:12	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	362,880	598,041	1.6480
01 Feb 2013 18:26:38	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	336,960	554,894	1.6468
30 Jan 2013 03:32:29	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	311,040	511,479	1.6444
26 Jan 2013 05:14:21	1064622	15525849	hadcm3n_o2cu_2140_40_008270392_3	285,120	468,520	1.6432