A system to quantify industrial data quality

Abstract

The digital universe is expanding at an exponential rate { both in size and data variety. This phenomenon, known as big data, is making an impact on all industries. By adopting the big data principle, organisations can become more profitable, and efficient, and deliver better services and products. The increase in the volume of data, however, also leads to an increase in factors that could lead to poor data quality. Poor data quality has been shown to increase operational costs, decrease customer satisfaction, and lead to inefficient decision-making processes. However, the benefits of managing and investing in data quality include increased customer satisfaction, increased revenue, reduced costs and greater confidence in analytical systems. There are only a limited number of existing systems to analyse industrial data quality. Most systems focus on customer relation management or healthcare applications. Very few complete systems are aimed at analysing industrial data. Therefore, a need exists to develop a method and system to quantify industrial data quality. During this study, an industrial data quality analysis method was developed to analyse industrial data quality. The method builds on the fundamentals of industrial sensor data analysis and data quality measurement methods. Furthermore, an automated system was developed to quantify industrial data quality using the results of the analysis method. The functionality of the analysis method was verified using scenarios that simulated common errors in industrial data. The method and system were implemented at an industrial company to prove the feasibility of a data quality analysis system. The system was validated using condition monitoring input data for a large deep-level gold mining operation in South Africa. Further analysis was performed using the condition monitoring input data. Results indicated that fans were the component with the most data quality problems, that temperature data was the measurement that had the poorest data quality, and that static data was the most common error found. Most of the data problems were traced back to poor communication between relevant systems. It was also found that the industrial company's condition monitoring data for the specific gold mining operation was approximately 16% erroneous. The most common errors found in the specific condition monitoring data were caused by static data, missing data and data that exceeded limits. The most likely cause for the missing data and static data is once again communication failures between relevant systems. The exceeds limits errors occur due to pieces of equipment being run outside their operational limits. The developed system met the objectives of the study and succeeded in quantifying industrial data quality. The accuracy of the results can be improved by expanding the analysis to include contextual knowledge. The design of the system also allows for a platform to which more metrics can be added as required, and a basis for the development of a cross-industry platform for quantifying industrial data quality.