Task 15550574

Name	hadcm3n_zk6j_1920_40_008281641_1
Workunit	8432776
Created	18 Jan 2013, 2:42:50 UTC
Sent	18 Jan 2013, 2:43:07 UTC
Report deadline	19 Apr 2013, 10:10:18 UTC
Received	20 Feb 2013, 7:44:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1237173
Run time	3 days 4 hours 17 min 54 sec
CPU time	2 days 9 hours 8 min 14 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	3.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 23:53:30 (191176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:07 (191092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:09 (183848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:48 (192076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:42 (191168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:58 (193156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:13 (193156): No heartbeat from core client for 30 sec - exiting 09:13:14 (193156): No heartbeat from core client for 30 sec - exiting 09:13:15 (193156): No heartbeat from core client for 30 sec - exiting 09:13:16 (193156): No heartbeat from core client for 30 sec - exiting 11:34:59 (194028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:35:02 (194028): No heartbeat from core client for 30 sec - exiting 11:35:03 (194028): No heartbeat from core client for 30 sec - exiting 11:35:04 (194028): No heartbeat from core client for 30 sec - exiting 11:35:05 (194028): No heartbeat from core client for 30 sec - exiting 11:35:06 (194028): No heartbeat from core client for 30 sec - exiting 11:35:07 (194028): No heartbeat from core client for 30 sec - exiting 13:07:41 (194644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:43 (193496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:39:25 (197136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:42 (196800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:23 (197728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:29 (197728): No heartbeat from core client for 30 sec - exiting 21:01:30 (197728): No heartbeat from core client for 30 sec - exiting 21:01:31 (197728): No heartbeat from core client for 30 sec - exiting 21:01:32 (197728): No heartbeat from core client for 30 sec - exiting 21:01:33 (197728): No heartbeat from core client for 30 sec - exiting 21:01:34 (197728): No heartbeat from core client for 30 sec - exiting 21:13:28 (197184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:41 (197344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:11 (190816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:30:25 (200116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=201448, iMonCtr=1 Model crash detected, will try to restart... 05:13:49 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:40 (588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:41 (588): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 06:46:26 (12280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:19:55 (12596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:00 (12596): No heartbeat from core client for 30 sec - exiting 07:20:01 (12596): No heartbeat from core client for 30 sec - exiting 07:20:02 (12596): No heartbeat from core client for 30 sec - exiting 07:20:03 (12596): No heartbeat from core client for 30 sec - exiting 07:20:04 (12596): No heartbeat from core client for 30 sec - exiting 07:20:05 (12596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 12:22:26 (12616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:22:36 (12616): No heartbeat from core client for 30 sec - exiting 15:30:49 (15220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:50 (15220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 09:13:42 (55352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:44 (55352): No heartbeat from core client for 30 sec - exiting 10:23:50 (59652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:29:53 (60228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:32 (56504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:33 (65360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:55:08 (65856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:55:13 (65856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:10:41 (112752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:49 (112752): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Feb 2013 05:32:25	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	207,360	216,265	1.0429
16 Feb 2013 14:55:34	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	181,440	189,607	1.0450
16 Feb 2013 06:23:27	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	155,520	162,763	1.0466
15 Feb 2013 21:35:55	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	129,600	135,705	1.0471
15 Feb 2013 12:35:03	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	103,680	109,157	1.0528
13 Feb 2013 17:21:11	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	77,760	82,343	1.0589
13 Feb 2013 04:00:22	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	51,840	54,939	1.0598
12 Feb 2013 16:16:30	1237173	15550574	hadcm3n_zk6j_1920_40_008281641_1	25,920	27,517	1.0616