Task 19963702

Name	wah2_sas50_kz98_201312_13_448_010719408_0
Workunit	10719408
Created	12 Sep 2016, 17:19:37 UTC
Sent	12 Sep 2016, 17:38:20 UTC
Report deadline	25 Aug 2017, 22:58:20 UTC
Received	28 Feb 2017, 0:38:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1188624
Run time	9 days 1 hours 15 min 51 sec
CPU time	5 days 0 hours 35 min 54 sec
Validate state	Invalid
Credit	6,859.15
Device peak FLOPS	3.71 GFLOPS
Application version	Weather At Home 2 (wah2) (region independent) v8.12 windows_intelx86
Peak working set size	412.99 MB
Peak swap size	348.16 MB
Peak disk usage	203.68 MB
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4836, selfPID=7828, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8956, selfPID=7080, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8784, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8876, selfPID=7068, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2352, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4280, selfPID=5400, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8908, selfPID=7208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6228, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6960, selfPID=5948, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3120, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6704, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8244, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1852, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6308, selfPID=6112, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8848, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7364, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3228, selfPID=6192, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5636, selfPID=5396, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8536, selfPID=6972, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5888, selfPID=1572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9028, selfPID=7588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9060, selfPID=7852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2832, selfPID=7596, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3984, selfPID=8132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6176, selfPID=4064, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3080, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8476, selfPID=4896, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7880, selfPID=5068, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3716, selfPID=2124, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3416, selfPID=7268, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7680, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8568, selfPID=5208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8232, selfPID=7432, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2768, selfPID=3180, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9176, selfPID=7200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6884, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3128, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8676, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:25:16 (4256): called boinc_finish(0) Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3972, selfPID=3044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4396, selfPID=8240, iMonCtr=1 Model crash detected, will try to restart... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5608, selfPID=4776, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6508, selfPID=4552, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5384, selfPID=3388, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6268, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5060, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7456, selfPID=5004, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1716, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2640, selfPID=5576, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1572, selfPID=4452, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3344, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=984, selfPID=4792, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7652, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7712, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7512, selfPID=4772, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1188, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8976, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8804, selfPID=4136, iMonCtr=1 Model crash detected, will try to restart... CGlobal Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6752, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5056, selfPID=3840, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1172, selfPID=6064, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1920, iMonCtr=2 CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8644, selfPID=7276, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8476, selfPID=6388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1196, selfPID=7944, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8412, selfPID=7792, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8436, selfPID=5644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7656, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3296, selfPID=5740, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5816, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6696, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7528, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7588, selfPID=5920, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8940, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8936, selfPID=1028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8308, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3940, selfPID=7852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5504, selfPID=6600, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8160, selfPID=7016, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8460, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8492, selfPID=5808, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7948, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5068, selfPID=7728, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9072, selfPID=5584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1688, selfPID=3504, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6944, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4284, selfPID=7028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4508, selfPID=6536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8324, selfPID=5560, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8380, selfPID=7468, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5532, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8952, selfPID=6460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8276, selfPID=6908, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_sas50_kz98_201312_13_448_010719408/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_sas50_kz98_201312_13_448_010719408/dataout/region_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/xadae.pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/xacxf.pipe_dummy 2048 Leaving CPDN_Main::Monitor... 11:32:56 (7364): called boinc_finish(0) </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>wah2_sas50_kz98_201312_13_448_010719408_0_r1573868275_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>wah2_sas50_kz98_201312_13_448_010719408_0_r1573868275_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>wah2_sas50_kz98_201312_13_448_010719408_0_r1573868275_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>wah2_sas50_kz98_201312_13_448_010719408_0_r1573868275_13.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>wah2_sas50_kz98_201312_13_448_010719408_0_r1573868275_14.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Jan 2017 02:16:01	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	103,979	412,576	3.9679
14 Jan 2017 23:51:03	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	92,459	375,845	4.0650
07 Jan 2017 21:11:57	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	80,939	327,939	4.0517
01 Jan 2017 00:43:22	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	69,419	281,255	4.0516
14 Oct 2016 20:57:15	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	57,899	233,052	4.0251
03 Oct 2016 01:43:52	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	46,379	186,112	4.0129
23 Sep 2016 03:51:00	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	34,859	139,742	4.0088
19 Sep 2016 01:33:11	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	23,339	93,853	4.0213
15 Sep 2016 22:50:56	1188624	19963702	wah2_sas50_kz98_201312_13_448_010719408_0	11,819	47,606	4.0279