Automate Transform Analyze

Transcription

1 Competitive Intelligence 2.0 Turning the Web s Big Data into Big Insights Automate Transform Analyze

2 Introduction Today, the web continues to grow at a dizzying pace. There are more than 1 billion websites out there, containing on everything under the sun. The web is an enormous resource for intelligence professionals, providing a treasure trove of publicly available for collection and analysis. However, the web s big data is a two-edged sword. The good news is that there is more out there than ever before. That s also the bad news. Not only is there more data, it s also changing at an ever-increasing rate. Companies need to move in web time to keep abreast of their competitors, but traditional intelligence approaches cannot keep pace. Of course, any company can continue to collect the basics press releases, analyst reports, new product announcements and feedback from the field, for instance. However, that is missing the true point and potential of the web. There are so many hidden nuggets that can deliver profound insights if only they can be separated from the millions of pebbles in the stream. How can companies effectively mine the web for treasure and turn it into strategic insights and tactical tools? How do we create Competitive Intelligence 2.0, where we instantly distill vast amounts of content into actionable knowledge? A Hierarchical Model for Competitive Intelligence First, we need to redefine the role of the intelligence professional. Fortunately, this is a process that is already underway. A good intelligence analyst is worth their weight in gold, and the last thing they should be doing is scraping content from thousands of websites in a losing attempt to separate the wheat from the chaff. Instead, we need to think about intelligence in four layers: Content: undigested, unstructured data that exists on websites and other sources. Information: structured, precisely targeted data extracted from raw content. Knowledge: facts and patterns extracted from. Wisdom: insights and actions synthesized from knowledge. Think of this as a hierarchy with increasing value at progressively higher layers. In some ways, it is analogous to Maslow s hierarchy of human needs which starts out with basic physiological needs such as food, water and air, and then works its way up to self-actualization. Similarly, content is a basic physiological need for intelligence. However, it has no intrinsic value other than basic survival. It s only as we start to move up the hierarchy towards wisdom that we start to create increasing value. Wisdom Knowledge Information Content This is the goal of Competitive Intelligence 2.0. It uses technology to turn content the Web s big data into, and then uses technology again to turn this into knowledge. This frees intelligence professionals to focus primarily on two key areas creating wisdom, and identifying relevant new sources of content to feed into the intelligence machine.

3 From Content to Information The first step in Competitive Intelligence 2.0 is automatically turning content into. Superficially, this may seem to be a simple task, but in reality it is a major challenge with many dimensions. These include: Establishing clear intelligence goals Selecting the right web sources to support these goals 1. Establishing clear intelligence goals. 2. Selecting the right web sources to support these goals. 3. Choosing the right tools to extract web content. 4. Coping with changes in web source formats. 5. Turning content into usable. 6. Formatting and distributing. Let s look at each of these in turn. Formatting and distributing Turning content into usable From Content to Information Coping with changes in web source formats Choosing the right tools to extract web content Establishing Clear Goals Automating web content extraction is not an ad hoc process it s critical to clearly establish objectives up front. For example, is the goal to continuously adjust pricing to match competitors? Or, is the goal to track competitors product and market strategies? Perhaps the goal is to understand emerging weaknesses in a competitor s supply chain or even to look beyond direct competitors, scanning the horizon for emerging threats. Selecting the Right Web Sources Each set of intelligence goals dictates a different content acquisition strategy. For instance, adjusting pricing in the consumer products market involves monitoring competitors online distribution channels. On the other hand, monitoring job boards can give an early warning of competitors starting to invest in new product areas. These are only examples the goal is to choose web sources that align with your specific objectives. Establishing clear intelligence goals Selecting the right web sources to support these goals Choosing the Right Extraction Tools There are three basic choices when extracting content from web sources web scrapers, custom scripts and web content extraction platforms. Web scrapers are generally a poor choice they extract the entire contents of webpages, simply creating a copy of the content without structuring it or targeting precise fields. Web scrapers also cannot handle dynamic content for example, they cannot automatically enter search terms into sites or deal with things such as infinite scroll webpages. Custom scripts and web content extraction platforms such as Connotate provide much more comprehensive capabilities, including the ability to target precise content on a web page, as well as navigating complex, dynamic sites. However, scripts are extremely expensive to create. They have to be programmed, which can take several days for a single web source and even more if that source Choosing the right tools to extract web content

4 is complex. Web content extraction platforms, on the other hand, allow users to create extraction agents in a couple of hours or less, using visual tools to specify the content to be extracted. Generally, if there are more than 25 web sources to be monitored, then a web content extraction platform is a far more economical choice. Coping with Web Source Format Changes Web source format changes are another key thing to consider when choosing an extraction tool. Typical web sources change format once or twice a year. Scripts usually break when the format changes, leading to high maintenance costs. It s not unusual to need one full-time programmer for every 200 scripts. Web content extraction platforms, on the other hand, cope with most format changes. This typically means that one non-technical resource can maintain 5,000 extraction agents a 25x reduction in maintenance effort compared to scripts. Turning Content into Usable Competitive Information Once content has been extracted, it then needs to be transformed into that can actually be used. This involves a number of different steps, which can be accomplished using the same web content extraction platform. First of all, the data needs to be deduplicated so that there is only one copy, even if the same is collected from multiple sources. Similarly, the data needs to be inspected for changes, so that only new is gathered. Then, the data needs to be filtered to eliminate irrelevant, and then normalized so that similar content from disparate sources is transformed into a common structure for further analysis. Coping with changes in web source formats Turning content into usable Formatting and Distributing Competitive Information Finally, the needs to be distributed to downstream targets. In some cases, this means formatting it directly for human consumption. A good example of this is creating a news feed for sales or marketing teams. In other cases, the data needs to be formatted for automated downstream analysis for instance, the may need to be inserted into a data warehouse, or converted to a specific XML format for a particular downstream processing system. Formatting and distributing From Information to Knowledge Simply collecting is not enough. Too many organizations simply gather huge repositories of data, and then do nothing with it. In fact, Harvard Business Review reported that a 2016 survey showed that only half of companies actually use the intelligence they collect. The truth is that intelligence that sits on the shelf has no value it has to be analyzed and transformed into knowledge to deliver benefits. However, the sheer volume of collected from the web can be overwhelming. It becomes prohibitively expensive and time-consuming to analyze all of this manually and, because of the time this takes, it is often too late to act on the knowledge that emerges. The data also often contains hidden patterns that are difficult for humans to uncover, so that major knowledge goes undiscovered.

5 The goal of Competitive Intelligence 2.0 is to use technology to analyze as much of this data as possible, and to present the resulting knowledge in a way that is easily digestible by humans. Here are some powerful ways to transform into knowledge. Business Intelligence/Artificial Intelligence Business intelligence tools are designed to analyze vast amounts of data and that includes. Big data analytics is designed to uncover relationships in data, distilling vast amounts of into facts, trends and insights. Many business intelligence tools incorporate artificial intelligence techniques to extract meaningful knowledge for example, analytics tools use techniques such as Bayesian networks to uncover patterns in data, with the added advantage that they only report a pattern once it has passed a confidence threshold. This has many clear applications for example, monitoring consumer sentiment for products and services, or uncovering emerging trends. The other key advantage is that there is no need to tell these tools exactly what to look for they discover patterns by themselves, rather than just validating specific patterns provided to them by an analyst. Smart Data By combining collected with commercially available data sets, organizations can unlock unique insights that are not apparent from the alone. This is often referred to as smart data linked data sets that together generate new knowledge that wasn t previously available. Here is an example. One company wanted to gain insights into the health of a global manufacturer s supply chain. They started by mapping the manufacturer s supply chain. They then added on the various vendors within that supply chain, including things such as their location, financial performance, and so on. This gave the company a smart view of the engine that drove the manufacturer s ecosystem, allowing them to understand its strengths and weaknesses. They then combined this smart data with weather, so they could understand the impact of major weather events on the manufacturer s supply chain around the world allowing them to predict the manufacturer s future performance based on current environmental data. Conclusion The web is an enormous source of content content that can be transformed to provide profound advantage. However, the sheer volume of on the web is overwhelming it far exceeds the capacity of traditional intelligence approaches. The risk is clear intelligence professionals are increasingly likely to waste their time collecting huge volumes of content from the web, rather than focusing on high-value activities such as formulating strategies. Competitive Intelligence 2.0 offers a compelling way forward. It automates the collection of content, transforms content into structured, and then analyzes that to generate knowledge. By doing this, it unlocks the true CI potential of the web, and maximizes the value that intelligence professionals deliver.

6 Learn more: Contact us at or visit