Data Preparation for Enhancing Modern BI: Common Design Patterns

Transcription

1 Data Preparation for Enhancing Modern BI: Common Design Patterns

2 Introduction If you want to be successful in analyzing big data to mine it for insights that will move the needle for your business, then the data you re analyzing needs to be clean, organized, detailed and well-understood. Raw data lacking the necessary pedigree for analysis can be fixed and shaped the process of doing so is often referred to as data preparation. In practice, data preparation involves several subtasks. The most common ones are data cleansing and data transformation. Yet there are other subtasks, such as looking inside the data for other data of interest. Enriching data by joining it to other data sets, whether from internal corporate systems or external, publicly available data sets, is another way to discover the data hidden within the apparent data. There is a large laundry list of use cases for data preparation potentially as large as the candidate data sources multiplied by the ways in which the data will be analyzed. Many drive very unique requirements. For example: You re using a processing series of texts and wish to do lexical analysis to understand how people of different eras exhibited their attitudes through vocabulary. The analysis involves social media posts including status messages, tweets, comments and other responses to gauge mood and sentiment, or for mentions of a specific person or company or theme. You re applying geographical or geospatial content to perform garden-variety address matching and address correction, or to look at the number of customers in a geographical area defined by physical radius, drive time, ZIP code, congressional district or other mapped region. You want to look at purchasing patterns overtime, break your customers down into demographic groupings, and correlate shopping activity at brick and mortar locations with weather information. No matter what your analytic use case is, taking the data you have and changing it to the data you need is a critical step in the analysis. PAGE 2

3 Sentiment Analysis Name, Company Mentions Fraud Detection Data Preparation Weather Impact Geographical Slicing Figure 1: Data Preparation Driving Various Analysis Use Cases Click Stream Processing Data Cleansing As the need for analysis created the desire to take data offline and explore it in spreadsheet and desktop database applications, most datasets came in as extracts from transactional systems. The data in the originating systems were not meant for analysis and data discovery, and because of that, the data wasn t always so clean. When all a system needs to do is capture enough information to get a transaction out, the integrity of ancillary data may suffer. For example: Duplicate customer records (with slightly different spellings) were pretty common Field names were often a combination of capital letters and underscores, and may also have been very short, making the extracts far from self-documenting Extraneous characters could enter pretty much anywhere, and go uncorrected Even the crucial data, such as line item amounts, could be incorrect, due to data entry errors, illegible source documents, or other sources of error Because of this, data cleansing became a common step prior to analysis. Features such as de-duping, address correction, field renaming and data value anomaly detection PAGE 3

4 have become common place. Performing these tasks easily, quickly and with as much automated assistance as possible, is an important aspect of data preparation, and ETL, data warehousing and OLAP readily address data preparation requirements for cleansing. These tools work on the notion of combining a fact table (containing important numerical data) with several dimension tables (which provide drill-down categories). These tables come directly from operational databases they involve simple denormalizing of transactional tables, along with some level of aggregation. Data warehouses and OLAP subsist on data in that format and ETL is there to automate getting transactional data into it. Beyond Data Cleansing However, cleaning the data is a small part of the battle, and even a perfectly clean data set is often not ready for analysis. It takes strategy and experience to start with data in its raw format and pre-process it to work well for analysis. Typically, the data that leads to insights requires some interpretation and transformation of the raw data. This isn t necessarily a needle-in-a-haystack search it s more like an archeological dig, where you ll need to continually dust things off to get a better look. Big data analytics in particular involves a different process data discovery in which data is pulled from multiple sources, some of which are not transactional, then sifting through it until useable data can be found. That takes some constructivist thinking. Data Preparation is More Than Just Cleansing A relatively new set of products collectively characterized as self-service data prep tools has emerged in the market. These products are geared toward helping users of self-service BI tools fill a functionality gap to convert more complex data sets which are often very raw and untreated into organized formats the self-service BI tools can consume. However, with big data analytics there are a large number of use cases that have an expanded set of requirements that scale beyond what self-service data prep tools cover. Standard data transformation, cleansing and joins are a general part of data preparation, but there are additional needs such as: Looking inside the data for other data Very precise parsing Pre-aggregation of data, often in complex ways Shredding of hierarchical information into individual rows of data, or vice-versa Mining unformatted text for specific pieces of information Inserting value-added calculated or algorithmically derived fields PAGE 4

5 Preparation Is Integral to the Workflow It is important to also think outside the functionality scope and understand the role of data preparation in the larger analytic workflow. Keep in mind that the word preparation is somewhat a misnomer, as it implies that the data prep process takes place as an early step and from which one moves on. In big data discovery, data preparation may take place at any point in an iterative data discovery process. After analysis, the data discovery may reveal flaws in the data or the need to add new data, revealing new requirements for how the data should be shaped, interpreted, enhanced or cleansed. The diversity and amount of data available today is unprecedentedly large, creating data preparation requirements for big data analytics that expand beyond the scope of standard self-service data prep. The range of transformations you can perform on these vast datasets is also large and differs markedly from the relatively simple, formal process of extract, transform and load (ETL). PAGE 5

6 Data Preparation Design Patterns Data preparation involves exposing information in the data that is latent, obfuscated and/or which has a dependency on initial analysis before it can be discovered or detected. This need generates requirements well beyond the scope of most self-service data prep tools. Following are a number of common big data preparation design patterns that go beyond traditional cleansing, joining and transformation. These will make the promise of big data preparation more clear, more navigable and, in general, more actionable. By the time you ve finished reading this paper you should have a list of things that you want to get done with your data, a good set of ideas on how to proceed and a realistic expectation of the challenges you will likely encounter along the way. Sessionization Sessionization is capturing a number of discrete events into a specific window so that window can be analyzed as a unique entity. Sessionization is typically associated with clickstream analysis but is often used in time-series analysis as well. When it comes to clickstream analytics, while web log data can often be grouped and sorted by user and date, most clickstream analysis requires you to sort it by actual web browser session. This is done by carefully looking at the timestamp and then collating all log entries within a certain time window of each other. Sessionization is very powerful. This type of affinity grouping between otherwise independent time-series events makes all kinds of downstream analysis easy. You can also use the first and last time stamp in each group to calculate the length of the session, and count the number of rows in each group to get the number of clicks per session. Popular page pairs (i.e. two pages between which users often directly navigate) can be easily calculated as well. PAGE 6

7 Advanced grouping Sessionizing is a kind of custom grouping. Speaking more generally, custom grouping involves taking a large array of data rows and segregating them not by discrete values in a column but by some discretization of continuous values. Custom binning can be used across use cases, including fraud analysis, preventive maintenance, shopping patterns of walk-in customers at retail stores, and more. Abilities like custom binning, time windowing, statistical grouping (based, for example, on standard deviation) and path analysis functions are each relevant here. Does your data prep tool offer this functionality? If not, you may not be covered for important analyses. PAGE 7

8 Standard deviation and custom binning can be useful in applications like credit card fraud detection. Find all the transactions with amounts greater than two standard deviations from the average for transactions in the same vendor category and you have candidates for fraudulent transactions. Then cluster those records together in groups that represent, say, three-day transaction intervals, and you may see patterns of abuse that eliminate the false positives. Column splitting Column splitting refers to the process of breaking up columns that contain multiple pieces of information, into a series of smaller columns that contain a single value. Column splitting may be especially useful in the processing of data extracts from older transactional systems, some of which had tendencies to concatenate multiple values in the database and separate them only when populating fields on a screen. PAGE 8

9 Sometimes splitting a column can be done simply. Specify a delimiter character and instantly you have a number of new simple (non-compound), columns. Other times it s trickier, because the delimiters are inconsistent, and even consistent delimiters can be more complex than a specific single character. Some data prep tools make short work of the simple column splitting tasks: specify a compound column and a delimiter and new, separate, columns will be produced. Other products handle more complex splitting tasks. For example, breaking up free text into individual words or sentences, which may require expression-based separators, can help prepare data for sentiment analysis. Data Enrichment Data enrichment refers to the general task of integrating external data sets be they commercial, public or proprietary to derive more information from the data you already have. Enriching data is common in customer analytics where you want to see the impact of external factors (e.g. weather or median household income) on internally tracked data like customer spending levels. It can also be useful in building predictive models. Transactional history of who bought what and when is a great baseline for analysis but, in today s markets, organizations need to identify deeper insights from their transactional data. For example, we might want to ask, what was the impact of weather, seasonality, customer gender, income, or education level on sales? One can only find these insights through the process of data enrichment to the transactional data. PAGE 9

10 Many data prep products handle acquisition of public data sets, and some provide a graphical facility for blending or harmonizing them in with homegrown data. Some products will also handle complex joins between data sets, and precise ways to specify them. Column/row pivoting Pivoting refers to the process of turning data rows into columns, and vice versa. It comes up repeatedly with data sets like product catalogs, which track items, and a number of their attributes (like color or size). Attribute values often show up in rows, but are sometimes useful as classifications in which case modeling them as columns may work better for specific analyses. ID Item 1 Chair 2 Table 3 Coffee Maker ID Attribute 101 Color 102 Size 103 Material 104 Style Item Attribute Value ID Item Color Size Material Style 1 Chair Blue Large 2 Table Cherry Wood Shaker 3 Coffee Medium Maker Blue Large Cherry Wood Shaker Medium In relational databases, tables that have lots of columns, many of which are often empty, can be problematic. Such wide and sparse tables are not storage-efficient, and are sometimes reworked so that the would-be columns and their values are stored as pairs in a single, rather narrow table. Making these pseudo rows into real columns is necessary for most analytical scenarios. And, again, at other times, rows may contain actual values that, simply for analytical purposes, can work well as columns. For example, you might want each country in which you do business to be a single column, so that you can see the revenue for each. PAGE 10

11 Working with lists List processing involves constructing series of values into a single value, or extracting a particular value from such a series. Transforming a data set into lists can be useful in time-series analyses. Pulling values out can be useful when reading data from hierarchical file formats. In the Sessionization section, we discussed grouping web log data rows by session. In that scenario, concatenating visited pages makes it easy to determine popular landing pages and common exit pages. (This is discussed in more detail in the Path Construction section, below). The opposite type of transformation can also be helpful. Data stored in JavaScript Object Notation (JSON) format frequently contains a set of hierarchical child data as a list, which may need to be projected out into distinct rows, for further grouping and aggregation. In addition to processing all elements in a list, sometimes a particular list element, or set of elements, that is nonetheless a subset of the full list, needs to be extracted. Perhaps you d like to see just the members of a list that are also present in another list (useful when looking for commonalities or affinity); maybe you d like to remove null or empty members from a list, or remove all duplicate elements. You get the idea. You can perform these list processing functions in many products. The question to keep in mind is how much work you need to do to in the process. Do you need to perform these operations imperatively, by looping through all list elements, or is there a set of declarative functions such that each operation can be performed in a single step? Advanced parsing Parsing is the process of reading long text values and extracting smaller text passages, or individual values, from it. Breaking paragraphs into sentences, or sentences into words are examples. Crawling Web pages and looking for mentions of a particular phrase, or subsidiary data in a specific format, are others. The easier it is to parse text, the more parsing you can do. And the more complex the parsing patterns can be, the more powerful the resulting analyses. In general, parsing text, or encoding text into, and decoding text from, certain well-known formats, like HTML, XML and JSON, are important capabilities. And a technical standard, called regular expressions, provides a powerful way to find specific text, based on specifications of patterns, and extract that text, so that it can be analyzed. Known by developers as REGEX, it can be the key to converting raw data to refined data. PAGE 11

12 REGEX syntax is obscure. But an investment in learning it can bring very significant rewards. In addition, there are many websites that can provide the REGEX syntax you might need for a variety of use cases. For example, suppose you pulled down a web page, as HTML, containing names and US addresses of locations for an event. You know that to pull out the cities, states and zip codes for those event locations, you ll need to look for a proper name, followed by a comma, a space and a two letter abbreviation. That text will be followed another space and a 5-digit number (or a 9-digit number with a dash between digits 5 and 6). Imagine you re sifting through tweets and you d like to pull out all the hashtags you ve found. To do so, you can just use a regular expression (like \B#\w+ ) that specifies you want to see all words that begin with a # character. Some tools make advanced parsing easy. If your data preparation needs include advanced parsing such as using REGEX, you should ensure the tools you select offer this capability. PAGE 12

13 Path Construction Path analysis involves the manipulation of time series data to illustrate the points or values encountered, and the sequence in which they occurred. It is the very essence of clickstream, log, IoT and other analytics scenarios. Path analysis also builds on other techniques we ve discussed here, like using list processing and sessionizing your data. For example, to determine popular landing pages and common exit pages, start with simple web log data, then parse out the file name from the URL for each logged request. Next, sessionize the data and concatenate filenames from all the visited URLs into a list. That list will in fact describe a user s full path through your web site during their session. Now grab the first and last elements of each list to get the landing and exit page for each session. Combining these steps lets you see what landing pages visitors are gravitating toward, and also where you re losing them. Now, doing some profiling like determining the top 10 in each category is easy. This analysis is surprisingly simple, but without data preparation functionality for grouping and coalescing your data the right way, you wouldn t even get close. And while you want the data to be as clean as possible before you work with it, the mere cleansing of it is nowhere close to the whole story. In fact, it s just the very beginning. PAGE 13

14 Conclusion We ve articulated a number of use cases and patterns around data preparation, all of which involved operations that went well beyond data quality and remediation. Data preparation is not just about de-duplicating and removing corrupted or dirty data. It s also about re-shaping, or transforming, that data so the data reveals the answers to you. Cleansing data is important, especially in the big data world, where data is coming from unconventional sources. But a big part of data preparation, in its full scope, is to emancipate insight by reshaping data, and thus to make analysis more straightforward. The data preparation design patterns shown here cut across different analytics scenarios, and are industry-agnostic. Tokenizing text is useful for sentiment analysis, but it s also useful in path analysis. Enriching data works really well for marketing optimization, but is also applicable for scenarios ranging from figuring out how weather impacts equipment breakdowns to understanding event attendance patterns in different geographies. Ongoing Data Conditioning Perhaps the most important thing to understand about data preparation, especially when it comes to the design patterns discussed in the paper, is that it isn t just a oneand-done, up-front process. Instead, you can think of the whole discipline as data conditioning, which is an ongoing, iterative process. An analysis may need a curated, well prepared data set. Related analyses may require new datasets that further shape the data. This leads to analysts collaborating, sharing and building upon each other s prepared datasets, helping them save time and increasing their productivity. In addition, analytic pipelines, that run daily or weekly, might need to be visited over time, exposing the need for more data shaping, to get at information based on adjacent or more detailed results. Realizing you have additional data shaping needs after doing some analysis isn t a failure it s not even a matter of course correcting. In fact, it s a very natural workflow, and an indicator of well-directed discovery. As you analyze data, you become more familiar and intimate with it. As that happens, you better understand it, and you then understand the next question you want to ask. This is a cyclical process, and all of it takes place after data quality checks and cleansing have. To learn more about big data preparation, the key functionality required, and how it fits in with the larger big data analytics workflow, please visit the Datameer website at: PAGE 14

15 FREE TRIAL datameer.com/free-trial T WIT LINKEDIN linkedin.com/company/datameer 2016 Datameer, Inc. All rights reserved. Datameer is a trademark of Datameer, Inc. Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation. Other names may be trademarks of their respective owners.