Mastering New Challenges in Text Analytics

Technical report Mastering New Challenges in Text Analytics Making unstructured data ready for predictive analytics Table of contents Introduction.......................................................................................................................... 2 What is text analytics and how is it used? ............................................................................. 3 Approaches to understanding text ........................................................................................ 4 The SPSS Inc. text analytics process ..................................................................................... 5 Applying text analytics at the enterprise level ..................................................................... 17 Conclusion ......................................................................................................................... 17 SPSS Inc. products for text analytics ................................................................................... 18 About SPSS Inc. .................................................................................................................. 18 Appendix A: An explanation of some text analytics terms ................................................... 19 Appendix B: Algorithms used for assigning equivalence classes ......................................... 21 Appendix C: Examples of Text Link Analysis ........................................................................ 22 Additional reading on text analytics .................................................................................... 23 SPSS and PASW are registered trademarks and the other SPSS products named are trademarks of SPSS Inc. All other names are trademarks of their respective owners. © 2009 SPSS Inc. All rights reserved. MCTWP-0109 Introduction It’s no secret that the world has seen an explosion of information in the past 15 years, an explosion that experts predict will continue as the millions of people who use online resources continue to expand their usage, and the millions of people who do not yet have access to such resources gain it. Similarly, information stored as text in both business and government organizations has grown exponentially. To name just a few examples: n Opinion surveys are increasingly conducted online and results shared in real time n The boom in software applications supporting sales, customer service, or call center operations has led to massive amounts of text stored electronically in these applications’ notes fields n Technology analysts at IDC estimate that 62 billion e-mails are sent every day n Searchable Web sites generate enough information every day to fill millions of books n Web logs (blogs) and wikis, created by individuals and groups for personal and professional purposes are increasing exponentially: as of this writing, there may be more than 100 million blogs, with a new one created every second Such a vast expansion of the scale of global information exchange would have been almost unimaginable 40 years ago, when most business and government communications, as well as news reports and advertising, were paper-based. Yet it was 40 years ago that visionary researchers began to seek ways to enrich the knowledge of those working in medicine and other sciences, in government agencies, and in business by making it possible to uncover previously unknown connections in large collections of textual documents by using computer technologies. They created the discipline known as computational linguistics, which is now practiced at numerous universities and public and private research centers worldwide. Computational linguists initially focused their efforts on finding ways to categorize and explore concepts found in books, scholarly journals, legal briefs, patent applications, newspapers, reports, and other paper-based records that could be converted to digital formats. More recently, their efforts have expanded to include ways to “mine” the vast amount of textual information that is published digitally—online editions of newspapers, academic journals, and conference proceeding, for example. In addition, there is a wealth of content that originates in digital form—such as Web sites, blogs, wikis, e-mails, instant messaging (IM), as well as text embedded in forms, surveys, and in scientific, government, or corporate databases. There is a growing recognition that analyzing text has become essential in various types of scientific research, and that it adds significant value to other forms of data analysis, particularly when used to predict how people may act in certain situations. For example, in obtaining a well-rounded view of customer behavior, text analytics is critical because it provides insight into the nuances of attitudes and opinions that influence behavior. With the exponential growth of text in online formats, ways must be found to structure this information and make it available to researchers and decision makers. This paper briefly defines text analytics, describes various approaches to text analytics, and then focuses on the natural language processing techniques used by SPSS Inc.’s text analytics solutions. It concludes with descriptions of SPSS Inc.’s solutions for text analytics and their role in predictive analytics. 2 Mastering New Challenges in Text Analytics What is text analytics and how is it used? First, it may be helpful to clarify what we mean by the terms text analytics and predictive analytics. To clear up one misconception, text analytics is not the same as search. Search engines are a “top down” approach to finding information in textual material. This means that end users must know how to structure queries to arrive at exactly the desired information. Text analytics, by contrast, is a “bottom up” approach. It does not require users to know particular search terms. Instead, text analytics reveals the concepts and themes contained in a body of documents, and then maps the relationships between them. To provide a more formal definition: Text analytics is a method for extracting usable knowledge from unstructured text data through identification of core concepts, sentiments, and trends, and then using this knowledge to support decision making. A “document” might be a scholarly journal article, free text responses to a market research survey, records from a database—such as call center notes or customer e-mails—contents of a news feed, or even a crime scene report. Text analytics discovers connections and relationships not within a single document but across a large collection or “corpus” of documents. These connections and relationships can then be organized in ways that permit analysis either alone or in combination with other types of data. Practitioners of text analytics may use algorithms to describe clusters of concepts, or associations between certain concepts or named entities. Text analytics results can then be incorporated in models used for predictive analytics. Predictive analytics informs and directs decision making by applying a combination of advanced analytics and decision optimization to data, with the objective of improving business processes to meet specific organizational goals. Including textual or “unstructured” data along with the “structured” data found in databases or transaction records adds depth to the insights gained through data mining. Textual data often reveals attitudes and sentiments that, when combined with demographic or behavioral data, enable analysts to more reliably predict events, behaviors, or actions that individuals or groups are likely to engage in. Text analytics has been shown to deliver measurable benefits to organizations in a wide range of applications. For commercial organizations, these include: n Supporting improved customer relationship management (CRM) by providing a more well-rounded view of customers, their wishes and preferences, leading to more effective marketing, reduced churn, and improved customer loyalty and lifetime value n Catching the “voice of the customer” through surveys or data from Web 2.