Task 15801956

Name	hadcm3n_o3ji_2020_40_008373498_0
Workunit	8524357
Created	29 May 2013, 19:15:09 UTC
Sent	29 May 2013, 19:21:14 UTC
Report deadline	29 Aug 2013, 2:48:25 UTC
Received	30 Sep 2013, 17:41:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1229098
Run time	46 days 12 hours 58 min 53 sec
CPU time	43 days 2 hours 29 min 1 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	0.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4712, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6896, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:04:38 (5392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:10:50 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:59 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:23:12 (6264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:32:20 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:46 (6480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:10 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:28 (5828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:35 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5716, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dan87b0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dan87c0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dan87d0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dan87e0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dan87f0 Atmos Hold Restart file rename failed on atmos_restart.hold 09:44:41 (4576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3212, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:34:55 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:51 (6804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:56:43 (3732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:58:31 (7056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:08 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:15 (5500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:20 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:12 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:14 (2700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:13:57 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 16:27:08 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:27:43 (6268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4532, iMonCtr=1 Model crash detected, will try to restart... 08:20:38 (5336): Can't acquire lockfile (32) - waiting 35s 08:21:13 (5336): Can't acquire lockfile (32) - exiting 08:21:13 (5336): Error: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. (0x20) Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 20:00:32 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:21 (1760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:07:25 (4080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:14:35 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:14:36 (7124): No heartbeat from core client for 30 sec - exiting 22:14:38 (7124): No heartbeat from core client for 30 sec - exiting 22:15:54 (4788): No heartbeat from core client for 30 sec - exiting 22:15:56 (4788): No heartbeat from core client for 30 sec - exiting 22:15:57 (4788): No heartbeat from core client for 30 sec - exiting 22:15:58 (4788): No heartbeat from core client for 30 sec - exiting 22:16:00 (4788): No heartbeat from core client for 30 sec - exiting 22:16:01 (4788): No heartbeat from core client for 30 sec - exiting 22:16:02 (4788): No heartbeat from core client for 30 sec - exiting 22:16:04 (4788): No heartbeat from core client for 30 sec - exiting 22:16:05 (4788): No heartbeat from core client for 30 sec - exiting 22:16:06 (4788): No heartbeat from core client for 30 sec - exiting 22:16:07 (4788): No heartbeat from core client for 30 sec - exiting 22:16:09 (4788): No heartbeat from core client for 30 sec - exiting 22:16:10 (4788): No heartbeat from core client for 30 sec - exiting 22:16:11 (4788): No heartbeat from core client for 30 sec - exiting 22:16:13 (4788): No heartbeat from core client for 30 sec - exiting 22:16:14 (4788): No heartbeat from core client for 30 sec - exiting 22:16:15 (4788): No heartbeat from core client for 30 sec - exiting 22:16:16 (4788): No heartbeat from core client for 30 sec - exiting 22:16:18 (4788): No heartbeat from core client for 30 sec - exiting 22:16:19 (4788): No heartbeat from core client for 30 sec - exiting 22:16:20 (4788): No heartbeat from core client for 30 sec - exiting 22:16:22 (4788): No heartbeat from core client for 30 sec - exiting 22:16:23 (4788): No heartbeat from core client for 30 sec - exiting 22:16:24 (4788): No heartbeat from core client for 30 sec - exiting 22:16:26 (4788): No heartbeat from core client for 30 sec - exiting 22:16:27 (4788): No heartbeat from core client for 30 sec - exiting 22:16:28 (4788): No heartbeat from core client for 30 sec - exiting 22:16:30 (4788): No heartbeat from core client for 30 sec - exiting 22:16:31 (4788): No heartbeat from core client for 30 sec - exiting 22:16:32 (4788): No heartbeat from core client for 30 sec - exiting 22:16:34 (4788): No heartbeat from core client for 30 sec - exiting 22:16:35 (4788): No heartbeat from core client for 30 sec - exiting 22:16:36 (4788): No heartbeat from core client for 30 sec - exiting 22:16:38 (4788): No heartbeat from core client for 30 sec - exiting 22:16:39 (4788): No heartbeat from core client for 30 sec - exiting 22:16:40 (4788): No heartbeat from core client for 30 sec - exiting 22:16:41 (4788): No heartbeat from core client for 30 sec - exiting 22:16:43 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:16:44 (4788): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:16:55 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 11:07:38 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap0850 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap0860 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap0870 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap0880 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap0890 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on o3jika.dap08a0 Atmos Hold Restart file rename failed on atmos_restart.hold 11:34:19 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2013 16:41:54	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	777,600	3,724,136	4.7893
26 Sep 2013 08:47:13	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	751,680	3,635,214	4.8361
25 Sep 2013 09:28:56	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	725,760	3,556,658	4.9006
23 Sep 2013 11:37:31	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	699,840	3,403,893	4.8638
18 Sep 2013 10:03:43	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	673,920	3,325,752	4.9349
18 Sep 2013 10:03:43	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	673,920	3,325,752	4.9349
17 Sep 2013 06:37:25	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	648,000	3,245,646	5.0087
15 Sep 2013 05:48:10	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	622,080	3,127,603	5.0277
12 Sep 2013 18:57:50	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	596,160	2,986,873	5.0102
10 Sep 2013 06:03:24	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	570,240	2,916,758	5.1150
02 Sep 2013 09:59:37	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	544,320	2,822,721	5.1858
29 Aug 2013 21:41:18	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	518,400	2,628,063	5.0696
23 Aug 2013 19:54:13	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	492,480	2,449,275	4.9733
14 Aug 2013 16:08:57	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	466,560	2,298,226	4.9259
14 Aug 2013 16:08:57	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	440,640	2,197,141	4.9862
14 Aug 2013 16:08:57	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	414,720	2,137,054	5.1530
23 Jul 2013 21:35:11	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	388,800	2,011,869	5.1746
23 Jul 2013 19:40:04	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	362,880	1,820,355	5.0164
11 Jul 2013 09:56:27	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	336,960	1,671,916	4.9618
10 Jul 2013 07:40:37	1229098	15801956	hadcm3n_o3ji_2020_40_008373498_0	311,040	1,614,371	5.1902