The five components of a data management program: Data profiling, data quality, data integration, data enrichment and data monitoring

A DataFlux White Paper Prepared by: DataFlux Corporation Five Steps to More Valuable Enterprise Data Leader in Data Quality www.dataflux.com International and Data Integration 877–846–FLUX +44 (0) 1753 272 020 Data quality and Introduction data governance Every good business decision is based upon good data. Whether making operational, require a analytical or strategic decisions, every department in an organization from the boardroom combination of to the sales floor requires reliable, accurate information — from knowing which customers people, processes own what products, to knowing which customers are potential opportunities for up-sell — and technology. to make those decisions correctly. But it's also true that bad data inevitably leads to bad business decisions, improper strategies and poor customer service. As companies undertake data-dependent initiatives, such as enterprise resource planning (ERP), customer relationship management (CRM) or enterprise initiatives leading to complete data governance, it is important to remember that data quality issues exist in any organization. Successfully addressing these data quality issues — and establishing the controls to maintain the quality of that data — is critical to the success of any of these initiatives. There are many reasons for the explosion of inconsistent or unusable data, from simple human error to the lack of data standards across systems, business units or divisions. To overcome the problems of bad data, organizations need to create a technological foundation for finding and eliminating data quality problems. But data quality issues cannot be resolved by technology alone. Having the correct people and processes in place is also crucial. This paper will show how a five-phase process can successfully organize people, processes and technology in a proven data quality methodology that allows organizations to analyze, improve and control corporate data. 1 Data profiling, data quality, data The Five Steps to More Valuable Enterprise Data integration, data enrichment and Technology is only part of the solution to produce consolidated, high-quality corporate data monitoring information. While it's essential to have robust technology supporting any data quality are the five initiative, having an appropriate methodology for using that technology will be even more components of a essential to the success of the initiative. data management DataFlux has developed a five-component methodology for enterprise data quality and program . data governance based on over a decade of experience and thousands of implementations. The five components are data profiling, data quality, data integration, data enrichment and data monitoring. Together, these five steps provide a proven, practical approach to data governance that allows organizations to analyze, improve and control their data. Figure 1 - The five components of a data management program. These components provide a framework for the successful management of the entire data improvement process. When followed, this methodology results in a unified view of any type of data, including customer, product and supplier information anywhere in the enterprise. Data Profiling The first step in the process, data profiling is the examination of the structure, relationships and content of existing data sources to help create an accurate picture of the state of the data. By determining the current state of the data, data profiling helps in planning the best ways to correct or reconcile information assets. 2 There are three separate components of a data profiling exercise, each of which analyzes Data profiling data in a unique way, that together create a clear picture of the nature and scope of data provides an quality issues. accurate picture of the current state Data profiling is essentially the process of addressing key questions about the data for of corporate data. each of these components: • Structure discovery – Do the data patterns match expected patterns? Does the data match the corresponding metadata? • Data discovery – Are the data values complete, accurate and unambiguous? Is the data standardized according to established conventions? • Relationship discovery – Does the data adhere to specified required key relationships across columns and tables? Are there inferred relationships across columns, tables or databases? Is there redundant data? Figure 2 shows the type of report generated during data profiling efforts. This analysis is an example of data discovery, as it shows the number of null counts for different fields within the database. Note that the overall count of 5000 records is consistent, but some fields show a high number of null values. With an understanding that there are significant percentages of missing values in several key fields - ADDRESS, CITY, STATE, and particularly GENDER – business analysts can be immediately aware of the scope and nature of data quality issues. Figure 2 – This chart displays a high number of null values in the Sales table. 3 Profiling provides quantifiable information detailing the strengths and weaknesses of corporate data. This knowledge can be used as the basis for subsequent data improvement. The information gained from data profiling feeds the next phase - data The data quality quality. phase uses the knowledge gained Data Quality by profiling to The data quality phase applies the knowledge gained during data profiling to begin the address data process of building better data. This phase helps correct errors, standardize information, issues and begin and validate data throughout the enterprise. building better data. There are a number of different tactics employed during this phase. Some of the most commonly employed measures include: • Data standardization to address and correct multiple permutations of data. For example, ACME Manufacturing Corporation may be represented in the same data source as Acme Mftg Corp, ACME and ACME Manufacturing. Intelligent fuzzy matching allows these variants to be standardized to create a single nomenclature. • Pattern standardization to create valid patterns of data across tables and columns. Some pieces of data, such as phone numbers or Social Security Numbers, are found in easily recognized variant patterns. Others, such as product or item data, may have different standards across industries or companies. Pattern standardization can take information in non-standard formats and transform it into an accepted standard format. • Address verification to confirm that addresses are valid and actionable. For example, DataFlux software can determine if the city and state do not match the ZIP code. The software searches postal data, locates the proper ZIP code and changes it to create an accurate address. Figure 3 shows how technology can help create a step-by-step business rules to address data quality problems. In this example, the workflow starts with the source table, then progresses through standardization, address verification and other routines. The earlier data profiling routine uncovered these issues, and the user creates steps to address each identified problem. 4 Investigating and addressing the causes of poor data quality is essential. Figure 3 – A sample data quality workflow. The specific approach taken may differ for each data element, and the decision on the approach falls to the business area responsible for the data. It is also important to note that the data quality activity improves the existing data but does not address the root cause of the data problems.