Task 15485699

Name	hadcm3n_3ddm_1940_40_008258196_0
Workunit	8413320
Created	20 Dec 2012, 10:00:49 UTC
Sent	20 Dec 2012, 10:01:01 UTC
Report deadline	21 Mar 2013, 17:28:12 UTC
Received	12 Mar 2013, 9:44:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1157053
Run time	23 days 16 hours 6 min 5 sec
CPU time	21 days 17 hours 33 min 3 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4292, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5580, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4816, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5308, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4752, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4588, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5044, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4692, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/3ddmko.pjg6c10 Error converting file to netcdf: dataout/3ddmko.pig6c10 Error converting file to netcdf: dataout/3ddmko.pfg6c10 Error converting file to netcdf: dataout/3ddmka.phg6c10 Error converting file to netcdf: dataout/3ddmka.pgg6c10 Error converting file to netcdf: dataout/3ddmka.peg6c10 Error converting file to netcdf: dataout/3ddmka.pdg6c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5276, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5588, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4560, iMonCtr=1 Model crash detected, will try to restart... 10:59:23 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4152, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77326E5F read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ddm_1940_40_008258196/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Mar 2013 08:02:52	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	1,036,800	1,877,569	1.8109
06 Mar 2013 12:06:31	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	1,010,880	1,831,349	1.8116
04 Mar 2013 11:04:08	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	984,960	1,785,835	1.8131
03 Mar 2013 03:09:36	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	959,040	1,740,155	1.8145
27 Feb 2013 23:40:17	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	933,120	1,693,788	1.8152
26 Feb 2013 15:15:46	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	907,200	1,648,056	1.8166
25 Feb 2013 13:01:14	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	881,280	1,601,572	1.8173
23 Feb 2013 13:50:20	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	855,360	1,555,289	1.8183
21 Feb 2013 15:05:40	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	829,440	1,509,112	1.8194
19 Feb 2013 11:43:53	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	803,520	1,462,345	1.8199
17 Feb 2013 18:50:14	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	777,600	1,416,314	1.8214
17 Feb 2013 05:41:22	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	751,680	1,370,360	1.8231
16 Feb 2013 07:28:40	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	725,760	1,323,504	1.8236
15 Feb 2013 11:22:37	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	699,840	1,276,934	1.8246
12 Feb 2013 15:36:17	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	673,920	1,230,850	1.8264
10 Feb 2013 13:20:14	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	648,000	1,185,690	1.8298
09 Feb 2013 17:51:32	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	622,080	1,140,693	1.8337
08 Feb 2013 10:52:49	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	596,160	1,094,627	1.8361
07 Feb 2013 13:05:30	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	570,240	1,049,363	1.8402
06 Feb 2013 17:21:32	1157053	15485699	hadcm3n_3ddm_1940_40_008258196_0	544,320	1,002,635	1.8420