Task 12991261

Name	hadcm3n_s5q8_1940_40_007300276_1
Workunit	7497700
Created	20 Jun 2011, 22:42:14 UTC
Sent	20 Jun 2011, 22:42:27 UTC
Report deadline	20 Sep 2011, 6:09:38 UTC
Received	2 Jul 2011, 5:03:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1000477
Run time	5 days 12 hours 29 min 50 sec
CPU time	4 days 23 hours 57 min 8 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 18:00:14 (6488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:15 (6488): No heartbeat from core client for 30 sec - exiting 19:31:20 (6824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:22 (6824): No heartbeat from core client for 30 sec - exiting 19:32:38 (8412): No heartbeat from core client for 30 sec - exiting 19:32:40 (8412): No heartbeat from core client for 30 sec - exiting 19:32:41 (8412): No heartbeat from core client for 30 sec - exiting 19:32:42 (8412): No heartbeat from core client for 30 sec - exiting 19:32:43 (8412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:37 (7504): No heartbeat from core client for 30 sec - exiting 19:34:44 (7504): No heartbeat from core client for 30 sec - exiting 19:34:45 (7504): No heartbeat from core client for 30 sec - exiting 19:34:47 (7504): No heartbeat from core client for 30 sec - exiting 19:34:48 (7504): No heartbeat from core client for 30 sec - exiting 19:34:49 (7504): No heartbeat from core client for 30 sec - exiting 19:34:50 (7504): No heartbeat from core client for 30 sec - exiting 19:34:51 (7504): No heartbeat from core client for 30 sec - exiting 19:34:52 (7504): No heartbeat from core client for 30 sec - exiting 19:34:53 (7504): No heartbeat from core client for 30 sec - exiting 19:34:54 (7504): No heartbeat from core client for 30 sec - exiting 19:34:55 (7504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:56 (7504): No heartbeat from core client for 30 sec - exiting 19:36:17 (9716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:39:24 (9776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:26 (7608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:38 (8656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:28 (8948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:29 (8948): No heartbeat from core client for 30 sec - exiting 19:06:09 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:17:28 (684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:45 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:54:23 (8324): No heartbeat from core client for 30 sec - exiting 11:54:24 (8324): No heartbeat from core client for 30 sec - exiting 11:54:25 (8324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:01:43 (9620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:37 (2744): No heartbeat from core client for 30 sec - exiting 12:02:38 (2744): No heartbeat from core client for 30 sec - exiting 12:02:39 (2744): No heartbeat from core client for 30 sec - exiting 12:02:40 (2744): No heartbeat from core client for 30 sec - exiting 12:02:41 (2744): No heartbeat from core client for 30 sec - exiting 12:02:42 (2744): No heartbeat from core client for 30 sec - exiting 12:02:43 (2744): No heartbeat from core client for 30 sec - exiting 12:02:44 (2744): No heartbeat from core client for 30 sec - exiting 12:02:45 (2744): No heartbeat from core client for 30 sec - exiting 12:02:46 (2744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:37:25 (4616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:05 (7380): No heartbeat from core client for 30 sec - exiting 14:38:06 (7380): No heartbeat from core client for 30 sec - exiting 14:38:07 (7380): No heartbeat from core client for 30 sec - exiting 14:38:08 (7380): No heartbeat from core client for 30 sec - exiting 14:38:09 (7380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:10 (7380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1752, selfPID=1752, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s5q8ko.pje9c10 Error converting file to netcdf: dataout/s5q8ko.pie9c10 Error converting file to netcdf: dataout/s5q8ko.pfe9c10 Error converting file to netcdf: dataout/s5q8ka.phe9c10 Error converting file to netcdf: dataout/s5q8ka.pge9c10 Error converting file to netcdf: dataout/s5q8ka.pee9c10 CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2011 03:17:30	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	233,280	431,202	1.8484
01 Jul 2011 11:54:56	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	207,360	384,633	1.8549
30 Jun 2011 17:49:36	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	181,440	338,230	1.8641
28 Jun 2011 19:20:39	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	155,520	292,299	1.8795
24 Jun 2011 10:02:26	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	129,600	244,054	1.8831
23 Jun 2011 17:53:13	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	103,680	195,786	1.8884
23 Jun 2011 01:24:27	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	77,760	146,753	1.8873
22 Jun 2011 07:06:34	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	51,840	98,483	1.8997
21 Jun 2011 14:28:32	1000477	12991261	hadcm3n_s5q8_1940_40_007300276_1	25,920	49,118	1.8950