Lixto Online Market Intelligence

Reference Code: TA001439BIN Publication Date: July 2008 Author: Angela Eager, Somak Roy, and Karthik Balakrishnan TECHNOLOGY AUDIT Lixto Online Market Intelligence Lixto BUTLER GROUP VIEW ABSTRACT Lixto’s Online Market Intelligence solution, which includes vertical offerings for the travel, consumer goods, and electronics industries, provides an automated way of extracting (primarily pricing data) from Web sites and repurposing it for notification, analytics, and dashboard reporting. Companies need to respond quickly to changes in competitor offerings but traditional market research methods cannot meet the speed and data coverage requirements of the online world. Lixto provides modules for visually defining data patterns within Web sites, extracting the data, and performing data analytics. Butler Group is impressed with the solution’s visual data-type definition process which allows non technical users to quickly build up an exhaustive list of sources to be monitored. Also, the managed services option and Lixto’s domain focus lead to a reliable and low maintenance solution well aligned with the needs of the target sector. KEY FINDINGS Highly visual data pattern definition. Provides industry-specific reporting, analytics and visualisation. Automatic data collection and aggregation Heuristics-based data extraction handles from large numbers of external Web sites. minor page changes and AJAX-based sites. Requires manual configuration of Each data source requires manual connectors for each data source. navigation to identify data patterns. Based on XML, SOAP, and J2EE. Utilises supervised learning approach. LOOK AHEAD Lixto is planning to add improved quality control mechanisms in order to validate gathered market data. Lixto – Lixto Online Market Intelligence Published 07/2008 © Butler Group. This Technology Audit is a licensed product and is not to be photocopied Page 1 TECHNOLOGY AUDIT FUNCTIONALITY Over the past decade the Internet has evolved to become a formidable vehicle of commerce, with a rich ecosystem of transactional Web sites, and sites that provide comparative information related to vendors and products, across a wide range of consumer categories. The ubiquity and real-time nature of the Internet has enabled and driven the need for quick, and in certain cases even real-time responses, to competitors’ actions in areas such as product and price revisions, and customisations. This need for agility is particularly true for items with a finite life, such as hotel rooms and air line tickets. For these sectors, while the Internet presents an embarrassment of riches in terms of freely available information, there is a lack of solutions that offer a reliable and automated way of aggregating the free information in a structured enough manner, to enable these real time product and price changes. A few industries, most notably the airline industry, have developed shared platforms for real-time price and product information exchange, to streamline transactions. However, for most industries across sectors and geographies, aggregating freely available information from the Web in an automated, exhaustive, and reliable way, coupled with analytics capability, is the way to go for short term price and product revisions, and for long term strategy formulation. Product Analysis Lixto, through the company’s Online Market Intelligence solution, addresses the aforementioned problem. The solution allows enterprise users to create connectors for specific web sites that extract required data (such as hotel room rates) from various online sources, convert the unstructured data into a structured format, and finally repurpose and present the data to the end-user in a suitable format for notification, analysis, and integration with in-house business applications. The data collected is cleansed and aggregated to generate dashboard views and detailed reports. The solution for the hotel industry, which enables analysis of hotel room rates, is the focus for this Technology Audit. The solution can be used by travel industry players to gather competitor and market intelligence information from public Web sites; comparing prices to ensure their own pricing levels are appropriately set, for example. It can also be used to ensure pricing consistency within a distribution channel. In this scenario, a hotel could monitor its distribution channel Web sites to ensure that hotel room prices remain within a specified price range. Using the notification workflow capability, an alert can be configured in the event of a price change beyond a specified band. The detailed reporting capability can be used for analysing availability and price per competitor or per category. Lixto Online Market Intelligence is able to deliver business value by automating and bringing scale to an otherwise manual task. Data extraction is based around data pattern recognition and Web page navigation. It involves drilling down into the Web page content structure, or more accurately the HTML tree structure, of the target Web site and specifying the class of data items, such as a list of prices to be extracted, in a highly visual way. The ‘class of data items’ is specified by establishing a set of rules (such as: select lines that begin with the word ‘Price’), and using the Visual Developer function that is included with the solution to select instances (such as highlighting five different pages areas that have prices of items), using mouse clicks rather than coding. Lixto – Lixto Online Market Intelligence Published 07/2008 © Butler Group. This Technology Audit is a licensed product and is not to be photocopied Page 2 TECHNOLOGY AUDIT The extraction rule formulation process runs in parallel with the Web page navigation specification process. The navigation specification process involves specifying link traversals, form fill ups, and user log ons. The aforementioned processes run in parallel with the process of developing the data structure for the extracted data. The data structure could be imported from an existing structure or from a standard structure such as Really Simple Syndication (RSS). The navigation specification, ‘class of data item’ selection, and data structure building is performed using the Lixto Visual Developer product (based on Eclipse and the Mozilla browser engine). At the time of data extraction, the Lixto solution performs a heuristic analysis to retrieve all the data items that most closely match the specified rules. Runtime data extraction is performed by the Lixto Transformation Server. The Transformation Server extracts data based on configurable events, such as data retrieval schedules. Retrieved data is cleansed and aggregated and can be repurposed in a variety of ways. The suite provides automated rate shopping and also offers end users a set of pre-defined, easily customisable reports and dashboards in order to provide an analytical overview of pricing trends. Reporting capabilities also include configurable alert notification and the option to interface with in-house business applications and business intelligence systems. Butler Group is impressed with the highly visual way Lixto handles the data selection specification process, which ensures that the solution is suitable for use by marketing personnel (rather than IT specialists). Partly due to the heuristic approach, the solution can handle Web pages built with Rich Web Application technologies such as Asynchronous Javascript and XML (AJAX) and can tolerate minor changes in the page structure, such as appearance or positioning of page elements. In the event of erroneous data reporting, the class of data item specification can be tweaked using the Visual Developer tool. The solution is provided as a managed service, and can to a large extent handle the Web page changes on its own. Overall, Butler Group is impressed with the solution and believes the Lixto approach would be relevant to companies in the hotel industry irrespective of size and regional focus. There are a few aspects that potential users should be aware of, however. Separate connectors have to be created and maintained for each data source which could become an issue if very large numbers of data sources are involved.