Taking Data Quality to the Enterprise through Data Governance

MARCH 2006 Taking Data Quality to the Enterprise through Data Governance By Philip Russom TDWI REPORT SERIES A 101communications Publication Research Sponsors Business Objects Collaborative Consulting DataFlux DataLever Firstlogic IBM Informatica Trillium Software TDWI REPORT SERIES MARCH 2006 Taking Data Quality to the Enterprise through Data Governance By Philip Russom Table of Contents About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 About TDWI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 About the TDWI Report Series . . . . . . . . . . . . . . . . . . . .2 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Research Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 The State of Data Quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Defining Data Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Best Practices for Improving the Quality of Data . . . . . .6 Evolving Perceptions, Realities, and Initiatives. . . . . . . .9 Liability and Leverage—A Case for Data Quality. . . . . . . . . 10 The Problems of Poor-Quality Data . . . . . . . . . . . . . . . . 10 The Benefits of High-Quality Data . . . . . . . . . . . . . . . . . 13 Data-Quality ROI and Budget . . . . . . . . . . . . . . . . . . . . . 14 Taking Data Quality to the Enterprise . . . . . . . . . . . . . . . . . . 15 The Scope of Data-Quality Initiatives . . . . . . . . . . . . . . 15 Most Data-Quality Trends Lead to Enterprise Use . . . . 17 Data Governance and Enterprise Data Quality . . . . . . . . . . . 19 The State of Data-Governance Initiatives . . . . . . . . . . . 19 Anatomy of Data-Governance and Stewardship Programs . . . . . . . . . . . . . . . . . . . . . . . . . .21 Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 www.tdwi.org 1 Enterprise Data Quality About the Author PHILIP RUSSOM is the senior manager of research and services at The Data Warehousing Institute (TDWI), where he oversees many of TDWI’s research-oriented publications, services, and events. Prior to joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research, Giga Information Group, and Hurwitz Group. He also ran his own business as an independent industry analyst and BI consultant, and was contributing editor with Intelligent Enterprise and DM Review magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at prussom@tdwi.org. About TDWI The Data Warehousing Institute (TDWI), a division of 101communications LLC, is the premier provider of in-depth, high-quality education and training in the business intelligence and data warehousing industry. TDWI is dedicated to educating business and information technology professionals about the strategies, techniques, and tools required to successfully design, build, and maintain data warehouses. It also fosters the advancement of data warehousing research and contributes to knowledge transfer and the professional development of its Members. TDWI sponsors and promotes a worldwide Membership program, quarterly educational conferences, regional educational seminars, onsite courses, solution provider partnerships, an awards program for best practices, resourceful publications, an in-depth research program, and a comprehensive Web site. About the TDWI Report Series This series is designed to educate technical and business professionals about new business intelligence technologies, concepts, or approaches that address a significant problem or issue. Research for the reports is conducted via interviews with industry experts and leading-edge user companies, and is supplemented by surveys of business intelligence professionals. To support the program, TDWI seeks vendors that collectively wish to evangelize a new approach to solving business intelligence problems or an emerging technology discipline. By banding together, sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues. Please contact Wayne Eckerson (weckerson@tdwi.org) to suggest a topic that meets these requirements. Acknowledgments TDWI would like to thank many people who contributed to this report. First, we appreciate the many users who responded to our survey, especially those who responded to our requests for phone interviews. Second, our report sponsors who diligently reviewed outlines, survey questions, and report drafts. Finally, we would like to recognize TDWI’s production team: Jennifer Agee, Bill Grimmer, Denelle Hanlon, Deirdre Hoffman, and Marie McFarland. This special report is the property of The Data Warehousing Institute (TDWI) and is made available to a restricted number of clients only upon these terms and conditions. TDWI reserves all rights herein. Reproduction or disclosure in whole or in part to parties other than the TDWI client, who is the original subscriber to this report, is permitted only with the written permission and express consent of TDWI. This report shall be treated at all times as a confidential and proprietary document for internal use only. The information contained in the report is believed to be reliable but cannot be guaranteed to be correct or complete. For more information about this report or its sponsors, and to view the archived report Webinar, please visit: www.tdwi.org/edqreport. ©2006 by 101communications LLC. All rights reserved. Printed in the United States. The Data Warehousing Institute is a trademark of 101communications LLC. Other product and company names mentioned herein may be trademarks and/or registered trademarks of their 2 www.tdwi.org respective companies. The Data Warehousing Institute is a division of 101communications LLC based in Chatsworth, CA. Executive Summary Executive Summary Data quality is difficult to comprehend in its entirety, because of the diverse aspirations and actions Data quality is a collected under its broad umbrella. This includes standard technology and business practices that complex concept that improve data, like name-and-address cleansing, record matching and merging, house-holding, de- encompasses many duplication, standardization, and appending third-party data. Some of these tasks can be automated technologies and with software, while others—like entering data properly—are purely matters of business process. business practices. Given this complexity, it’s no wonder misconceptions abound, like thinking data quality is a one-time action that results in perfection. To the contrary, data quality is a complex concept that encompasses many data-management techniques and business-quality practices, applied repeatedly over time as the state of quality evolves, to achieve levels of quality that vary per data type and seldom aspire to perfection. Of the organizations TDWI surveyed, 82.5% continue to perceive their data as good or okay. Most business people However, half of the practitioners surveyed warn that data quality is worse than their organization think their data quality realizes, which explains why the number of organizations with a data-quality plan doubled between is good, while half of 2001 and 2005. Many companies took action on data quality because compliance provided a swift technical people say it’s kick in the pants. Other kicks came from initiatives for business intelligence, customer service, worse than perceived. global supply chain, and IT system consolidations and migrations. Two-thirds of respondents have studied the problems of data quality, while less than half have Data quality is driven studied its benefits. This indicates clearly that data quality initiatives are driven more by liability more by liability than than leverage.