USE TEXT ANALYTICS TO ANALYZE SEMI-STRUCTURED AND UNSTRUCTURED DATA

Transcription

1 USE TEXT ANALYTICS TO ANALYZE SEMI-STRUCTURED AND UNSTRUCTURED DATA

2

3 Thank you for participating in a workshop at MicroStrategy World If you missed or did not finish an exercise and want to complete it after the conference, use this workbook and access the supporting files at microstrategy.com/world-2019-workshops *Workshop files will expire 4/30/2019

4 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Introduction The MicroStrategy Intelligence platform empowers organizations to: Analyze vast amounts of data. Offer tangible solutions to business queries. Build data visualizations. Enable users to share their insights anywhere and anytime. MicroStrategy s out-of-the-box connectors provide instant access 80+ data sources, including Twitter, further supporting the data needs of the Intelligent Enterprise. 4 MicroStrategy

5 MicroStrategy World 2019 Workshop Book This hands-on workshop is an introduction to MicroStrategy s data analytics and visualization capabilities. In this workshop, you will: Use MicroStrategy Desktop to connect to data sources and extract information for analyses. Search for and import relevant information found through data mining techniques. Visualize the data with a word cloud and use a bag-of-words model to find valuable patterns in text. Measure the impact of customer opinion through the use of sentiment analysis. Data Exploration: MicroStrategy Desktop MicroStrategy Desktop is a powerful data discovery tool that is used throughout the workshop to help you explore your data and answer key business questions. With a simple drag-and-drop interface, you can build compelling dossiers, and interactive and intuitive business intelligence dashboards, quickly and easily. MicroStrategy 5

6 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Requirements For this workshop, you need MicroStrategy Desktop downloaded to your computer. If you already have the necessary software for this course, skip to the Getting started section. If you have not downloaded the necessary software, follow the steps below, as this software is required to complete the workshop. If you did not bring a laptop with you, if your laptop permissions prohibit you from downloading software, or you are otherwise unable to complete the necessary software installations, team up with a neighbor for the remainder of the workshop. Workshop software setup Follow the steps below to download MicroStrategy Desktop on your Mac or PC. 1 Download MicroStrategy Desktop for Mac or Windows 64-bit at 2 Populate the fields and click Download Now. 5 6 MicroStrategy 3 Choose the appropriate link based on your computer for either Windows or Mac. For Windows 1 Extract the MicroStrategy Desktop zip file found in C:\Users\ 2019 MicroStrategy, Inc.

7 MicroStrategy World 2019 Workshop Book <yourusername>\downloads. 2 Open the folder MicroStrategy Desktop Double-click DesktopSetup.exe. Follow the steps to complete the installation. For Mac 1 Double-click the MicroStrategy Desktop zip file found in MacintoshHD:\ Users\<yourusername>\Downloads. 2 Double-click MicroStrategy_11.1_Mac.dmg. Click to agree with the license. 3 Drag the MicroStrategyDesktop.app in to the Applications folder and follow the steps to complete the installation. MicroStrategy Desktop is now ready to go. Leave Desktop open, as we will use it throughout the remainder of this workshop. Download and install R For a Windows machine, use For a Mac machine, use For Macs, you may also need to install the tibble library after installing R, via the steps are below. Exercise R Script Copy the SA_TidyText.r script provided by your instructor to the following location. Windows: C:\ProgramFiles\MicroStrategy\Desktop\R Integration Pack\ RScripts Mac: //Users/<username>/ Documents/ MicroStrategy R Integration Pack/ RScripts Install the tibble library (for Macs only) 1 Open R by clicking R on your desktop. 2 In the command line, type install.packages( tibble ) MicroStrategy, Inc. 6 MicroStrategy 7

8 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 3 If prompted for a location, select the first option and click Continue. 4 If prompted to install the package which needs compilation, type n and press Enter. The tibble package is now installed. Getting started The Intelligent Enterprise is the ultimate data-driven organization. The Application Architect in the Intelligent Enterprise guides his/her team in creating effective, easy-to-maintain applications using BI data. Analysts within the organization provide this data by evaluating information from data gathered from a variety of sources, including social media. As the Application Architect for your organization, you want to analyze Twitter data from your customers to gauge their attitudes towards your company. Gauging customer sentiment will provide useful insights on how the market views your company, as well as words frequently associated with your product. Exercise 1: Connect to and search Twitter MicroStrategy makes it easy to connect to external data sources like Twitter. This allows you to search Tweets in a feed and import them, along with other variables, into an Intelligent Cube for analysis. Connect to Twitter data 1 Open MicroStrategy Desktop. 2 Click New Dossier. 3 On the toolbar, click Add Data and then select New Data. MicroStrategy Web and Desktop allow you to quickly and intuitively connect to almost any data source. For this course, you will use sample data retrieved from a MicroStrategy- provided Twitter account. 7 8 MicroStrategy

9 MicroStrategy World 2019 Workshop Book 4 Click Twitter. 5 Log into Twitter using the following Twitter credentials: Username: mstrworkshop Password: m$tr2019 If asked for a two-factor authentication address, enter: workshop@microstrategy.com Alternatively, you can utilize your own Twitter credentials to complete this workshop. 6 Click Authorize app to authorize Desktop to access the account. 7 In the Import Data from Twitter window, type MicroStrategy. Keep the fetch at Click Search. Because this workshop connects to live Twitter data, which changes constantly, 2019 MicroStrategy, Inc. MicroStrategy 9

10 Use Text Analytics to Analyze Semi-Structured and Unstructured Data your data will differ from that shown below and in other images throughout this workshop. 9 Click Prepare Data at the bottom right, to review and clean up the data. A sample of the tweets and corresponding data points are displayed in the Preview window. Each tweet is represented as a separate row at the bottom of the window. Exercise 2: Create a multiform attribute in the Twitter data 9 10 MicroStrategy When you imported your data, each column in the table was imported as a separate attribute. An attribute is a category of data that is descriptive rather than measurable. The employee attribute can have the forms ID, last name, first name, address, and , for example. Creating a multiform attribute combines the different attributes into a single attribute. The Userid and Username attributes are forms of the same attribute. You want to 2019 MicroStrategy, Inc.

11 MicroStrategy World 2019 Workshop Book create a multiform attribute with these in order to roll up sentiment scores from the tweet level to the user level. Create a multiform attribute 1 Click All Objects View at the top left of the Preview window to see a more comprehensive view of your attributes and metrics, without the data preview. An attribute represents a business concept, such as year, country, employee, product, etc. Attributes provide the context for the measured data (metrics) that are the core of any data analysis. A metric represents a business measure or key performance indicator, such as revenue, profit, headcount, or probability of purchase. 2 Scroll down to the bottom of the attributes, so that you see Userid and Username. 3 Select the Userid and Username attributes. Right-click the selection and 2019 MicroStrategy, Inc. 10 MicroStrategy 11

12 Use Text Analytics to Analyze Semi-Structured and Unstructured Data choose Create Multiform Attribute. 4 In the New Attribute Name box, type User. 5 You want to see only the description, not the ID, so clear the Display Form check box for Userid. 6 Click Submit. 7 Click Close, then Finish to import your data into your dossier. 8 To save your dossier, click Save on the toolbar. 12 MicroStrategy

13 MicroStrategy World 2019 Workshop Book 9 Click Browse PC to save your dossier in your My Documents folder. 10 Name the dossier Twitter Sentiment Score. This file is now a shareable MicroStrategy file (with a.mstr extension). Exercise 3: Create a dossier using the imported Twitter data Sentiment analysis measures the opinions and attitudes of a writer s words towards a particular topic within a given text. Sentiment analysis is often used to analyze the attitudes of specific groups towards various people and events, extracted from social media sources, like blogs, surveys, and tweets. When this R script is deployed as a MicroStrategy metric, it needs a single input, which is a vector of text elements. Each text element is processed with the following steps: 1 Break each text element into the individual words that element contains. This process of separating a larger block of text into the words it contains is known as tokenization. 2 Remove stop words such as the, of, a, and other common parts of speech that are irrelevant when it comes to analyzing sentiment. 3 Handle words that include technology-related features such as URLs, hashtags (which begin with #), and users (which begin 4 Join the remaining words to the words in each tweet, capturing the sentiments and emotions that they tend to be associated with. 5 Sum the sentiment values associated with the text element that it contains. Analyze user sentiment To analyze the Twitter data, create an R metric that returns the positive or negative sentiment value for each tweet. This metric will also create two visualizations a word cloud and a sentiment score histogram. 1 The Datasets panel displays the imported data as a dataset named microstr Twitter. Scroll down in the Datasets panel to the metrics, displayed with the metric icon. Right-click any metric and select Create Metric. MicroStrategy 13

14 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 2 Select Switch to Formula Editor at the bottom left. 3 Remove the existing metric formula by clicking Clear on the right. 14 MicroStrategy

15 MicroStrategy World 2019 Workshop Book 4 Copy the Score Metric Expression provided by your instructor in the Supporting files (also shown below), then paste it into the Formula box. The expression tells MicroStrategy to execute an R script. RScriptU<_OutputVar=Score, _RScriptFile="SA_TidyText.R", _Params= "FileName='SA_mstr', PlotWordCloud=TRUE, PlotHistogram=TRUE, RemoveRetweets=TRUE, SaveCSV=TRUE">(Max(Text){~+}) Mac users need to specify the full file path to the R script. RScriptU<_OutputVar=Score,_RScriptFile="/Users/username/Desktop/SA_TidyT ext.r", _Params="FileName='SA_mstr', PlotWordCloud=TRUE, PlotHistogram=TRUE, RemoveRetweets=TRUE, SaveCSV=TRUE">(Max<UseLookupForAttributes=False>(Text){~+}) 5 In the Metric Name box, type Sentiment Score. 6 Validate the expression to ensure that the function is in the right syntax, by clicking Validate. MicroStrategy 15

16 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 7 Click Save to create the metric. The new metric is displayed in the Datasets panel. Use the new metric to create a grid 1 Drag the Text attribute from the Datasets panel to the Rows drop zone in the Editor panel. 2 Drag the Sentiment Score metric to the Metrics drop zone. Your grid, with updated data, should resemble the following Creating the sentiment score metric generated a word cloud and a sentiment score histogram. These visualizations will be added to your dossier as images. Generate a Word Cloud 1 Add a second page to your dossier by right-clicking Page 1 in the Contents panel and selecting Insert Page. Page 2 is added and displayed. 16 MicroStrategy

17 MicroStrategy World 2019 Workshop Book To provide a guided experience for end users, dossiers include a table of contents that helps you logically organize content into chapters and pages. 2 Insert an image into Page 2 by clicking Image. 3 In the image container, click Browse and navigate to: On Windows: C:\Program Files\MicroStrategy\Desktop\R Integration Pack\RScripts On Mac: /Users/<Username>/Library/Containers/ MicroStrategy.Workstation.DossierEditor/Data Note: if you cannot find the file path, it may be hidden. You will have to search SA_MSTR_WordCloud.jpg. 4 Select the image titled SA_MSTR_WordCloud and click Open. The image can also be found in the Supporting Files folder provided by your instructor. If you have any trouble opening the file, you can instead drag and drop the image from the folder into the Enter An Image URL or Choose A File box in the image container. 5 You want the word cloud visualization to expand to fill the entire page. Delete the blank visualization by clicking Menu on the title bar of Visualization 1 and selecting Delete. MicroStrategy 17

18 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Analyze the Word Cloud. Notice how the R script catered to the Twitter data by retaining useful punctuation such as symbol. Your word cloud may differ from the following image since different data is imported from Twitter. Generate a sentiment score histogram 1 To add a new image to Page 2, click New Image. 2 Use the same file path to access the sentiment score histogram image as you used to access the word cloud. Select the SA_MSTR_Histogram image. The histogram shows sentiment scores across the entire dataset of tweets. According to the histogram, the majority of tweets about the product are neutral, with many more positive than negative. Nevertheless, it is your job to uncover the user(s) who are negatively influencing sentiment. 18 MicroStrategy

19 MicroStrategy World 2019 Workshop Book 3 Drag and drop the Histogram of tblsents$score image below the word cloud, so that they display as shown below: Create a second metric You will now create another new metric to return the positive and negative sentiment score for each user, rather than each Tweet. 1 In the Contents panel, click Page 1 to access your grid visualization with tweet text and scores. 2 You want to create a sentiment score metric that will target users sentiment scores, rather than the sentiment score for individual tweets. In the Datasets panel, right-click any metric, and select Create Metric. MicroStrategy 19

20 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Data Analysis Text Analytics 3 At the bottom left of the Metric Editor New Metric window, select Switch to Formula Editor. 4 Remove the default metric formula by clicking Clear, on the right. 5 Copy the metric expression from the Score Metric Expression Supporting File folder provided by the instructor, as well as below. Paste the Expression into the Formula box. For Mac users, remember to include the full file path to the R script. RScriptU<_OutputVar=Score, _RScriptFile="SA_TidyText.R", _Params= "FileName='SA_mstr', PlotWordCloud=TRUE, PlotHistogram=TRUE, RemoveRetweets=TRUE, SaveCSV=TRUE">(Max(Text){~+}) 6 Update the expression syntax to target the User attribute, by replacing {~+} with {User}, as shown below. When the list of matching attributes and metrics is displayed, select User MicroStrategy 2019 MicroStrategy, Inc.

21 MicroStrategy World 2019 Workshop Book 7 In the Metric Name box, type User Sentiments. 8 Click Validate. 9 Click Save. The new metric is displayed in the Datasets panel. Create a grid using the User Sentiment Score metric 1 On the toolbar, click Insert Visualization. 2 Move Visualization 2 above Visualization 1. To do this, drag Visualization 2 s title bar above Visualization 1. Drop it when a blue line is displayed, as shown below: 3 Add the User attribute to the Rows drop zone in the Editor panel and the User 2019 MicroStrategy, Inc. 20 MicroStrategy 21

22 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Sentiments metric to the Metrics drop zone. 4 The User attribute displays the user name, rather than the user ID, as you set up when you created the multiform attribute. If you wanted to display the ID instead of, or as well as, the name, right-click the User column header in the grid, point to Display Attribute Forms, and select or clear the check boxes for the forms. 5 To rename the visualizations, double-click their title bars and rename them the following: Visualization 1 (on the bottom of the page): Tweet Sentiment Scores Visualization 2 (on the top of the page): User Sentiment Scores 22 MicroStrategy

23 MicroStrategy World 2019 Workshop Book Filter the tweet-level grid using the user-level grid You want to be able to quickly and easily see a specific user s tweets and their scores. To do this, create a filter on the User Sentiment Scores grid that targets the Tweet Sentiment Scores grid. 1 In the title bar for the User Sentiment Scores grid, click menu and choose Select Targets. User Sentiment Scores is labeled SOURCE. 2 Click the Tweet Sentiment Scores grid. It is labeled TARGET. MicroStrategy 23

24 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 3 Click Apply at the top right. 4 In the User Sentiment Scores grid, click any user to view their tweets and sentiment score. The Tweet Sentiment Scores grid displays only the tweets for the selected user. 24 MicroStrategy

25 MicroStrategy World 2019 Workshop Book 5 To display all the tweets in the Tweet Sentiment Scores grid, click the selected user again to clear the filter. 6 Save and close your dossier. You have helped your business locate social media users who hold negative perceptions about your company s product. You can use this information to directly reach out to users that may have had a negative experience and try to improve their future experiences. Exercise 4: Build a text analytics application The R metric allows you to tokenize, normalize, and remove stop words without needing to restructure the data through data wrangling. Your next task is to build a text analytics application to assess messaging, opinions, and sentiment from a recent Twitter campaign. Text analytics allow you to work with unstructured data like Tweets, utilizing data wrangling to mitigate discrepancies such as slang, different languages, and grammatical or spelling errors. Create a new dossier and import data 1 In Desktop, select New Dossier. 2 On the toolbar, click Add Data and then select New Data. MicroStrategy Web and Desktop allow you to quickly and intuitively connect to almost any data source. For this course, you will use sample data retrieved from a MicroStrategy- provided Twitter account. MicroStrategy 25

26 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 3 Click Twitter. 4 Log into Twitter using the following Twitter credentials: a. Username: mstrworkshop b. Password: m$tr2019 If asked for a two-factor authentication address, enter: workshop@microstrategy.com Alternatively, you can utilize your own Twitter credentials to complete this workshop. 5 Click Authorize app to authorize Desktop to access the account. 6 In the Import Data from Twitter window, type MicroStrategy and change Fetch to Click Search. Because this workshop connects to live Twitter data, which changes constantly, your data will differ from that shown below and in other images throughout this workshop MicroStrategy

27 MicroStrategy World 2019 Workshop Book 8 Click Prepare Data at the bottom right, to review and clean up the data. A sample of the tweets and corresponding data points are displayed in the Preview window. Each tweet is represented as a separate row at the bottom of the window. 9 Click All Objects View at the top left of the Preview window to see a more comprehensive view of your attributes and metrics, without the data preview. 10 Scroll down to the bottom of the attributes, so that you see Userid and Username. 11 Select the Userid and Username attributes. Right-click the selection and choose Create Multiform Attribute. 12 In the New Attribute Name box, type User. 13 You want to see only the description, not the ID, so clear the Display Form 2019 MicroStrategy, Inc. MicroStrategy 27

28 Use Text Analytics to Analyze Semi-Structured and Unstructured Data check box for Userid. 14 Click Submit. Process the data: Text analytics Natural language processing includes two key concepts: Tokenization is the parsing of content into smaller pieces, such as words or sentences. These are called tokens. Normalization, also called word stemming, is the process of reducing a group of words to its base or stem. For example, love, lovely, loved, and love, are all variations of the same stem, love. In the next few steps, you will process the data for analysis. To wrangle the data, a script file (datawrangle.txt) that has the wrangling steps has been provided. Creating a script of the data wrangling process is another useful MicroStrategy feature that you can utilize to save time when processing similar data. To utilize the data wrangling script provided in the supporting files folder proceed to the Wrangle data utilizing the script section. To wrangle the data manually, skip to the Wrangle data manually section. Wrangle data utilizing the script 1 Click Wrangle at the top of the Preview window to transform the data. 2 In the Script area of the Data Wrangling window, click Import MicroStrategy

29 MicroStrategy World 2019 Workshop Book 3 Click Browse at the bottom of the Import History Script window. 4 Navigate to the Supporting Files folder provided by your instructor, select the datawrangle.txt file, and click Open. 5 Click Apply. 6 Now that you have finished wrangling your data utilizing the script, skip to the Finish data import section. Wrangle data manually 1 Click Wrangle at the top left of the Preview window to transform the data. 2 In the Data Wrangling window, scroll through the Sample data area until you see the Text column. The columns are displayed in alphabetical order, from left to right. 3 Click the arrow at the top of the Text column and select Duplicate Column. 4 The new column is named Text 1. To rename the column, double-click the column header and rename it Tokens. 5 Extra white space is commonly added to tweets, both for creative reason and by mistaken. The first step in cleaning your data is to remove any consecutive white space. Select the Tokens column MicroStrategy, Inc. MicroStrategy 29

30 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 6 From the Select Function drop-down list at the top of the window, select Collapse Consecutive Whitespace. 7 We need to organize the data into one token per row. A token is a meaningful unit of text; in this case, each word is a token. Select the Tokens column. 8 In the Select Function drop-down list, select Split Cell Into Rows On Separator. 9 A new drop-down list is displayed, with Comma selected as the default separator. Select Whitespace from this drop-down list MicroStrategy 10 Click Apply at the top of the Data Wrangling window. The data wrangler 2019 MicroStrategy, Inc.

31 MicroStrategy World 2019 Workshop Book tokenizes the data into one word per row. Notice the data wrangling tool creates a script of all the actions that you perform on the top right. This script can be saved to your machine, uploaded at a later date, and applied to a new dataset. In the Script area at the top right of the data wrangling page, the steps you took are now part of the stored procedure. Apply an ID to each token The Tweet ID column is now missing an entry for many of the newly created rows. You need to associate each token with its original tweet ID. 1 In the Data Wrangling window, scroll to view the Id column. Double-click the Id column, and rename it Tweet ID. 2 Select the Tweet ID column. 3 From the Select Functions drop-down list, choose Fill Down so that each token corresponds to its Tweet ID. Normalize the Tokens column 2019 MicroStrategy, Inc. 30 MicroStrategy 31

32 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 1 Select the Tokens column. 2 From the Select Functions drop-down list, choose To Lowercase. Converting tokens to lowercase normalizes them, making it easier to compare to other data or to combine with other datasets. Now that you have the column separated into individual tokens and an ID applied to each, you can stem the words in the Text column. Cluster and edit the Tokens column 1 You now want to normalize the Tokens column with stemming. With the Tokens column selected, select Cluster and Edit from the Select Functions drop-down list MicroStrategy The clustering and edit uses a fingerprint algorithm, which groups similar terms into clusters. One cluster shows all variations of the word MicroStrategy that could be grouped under one unified stem. 2 To merge the rows into one stem, select the check box in the Merge column 2019 MicroStrategy, Inc.

33 MicroStrategy World 2019 Workshop Book for the row that displays variations of microstrategy in the cluster. This allows you to merge these variations under a common stem. 3 In the New Cell Value column for this row, type microstrategy. 4 If you have a row that includes punctuation marks, you should remove them from the tokens. As in the previous step, select the box in the Merge column for this row. In the New Cell Value column for this row, type delete. Note: Because the data will be different depending on when you go through this workshop, you may not have the symbols or words below. 5 Repeat the previous steps to complete the stemming process, until all the rows in the Cluster and Edit window are updated to meaningful words. Try it on your own or use the following table for reference to confirm the stem words that may need to be updated. You may have different words than those listed above. Change them (for example, by removing #, $, or other symbols) so that they are more meaningful and easy to understand. 6 Make sure that the check box is selected in the Merge column for each of the 2019 MicroStrategy, Inc. 32 MicroStrategy 33

34 Use Text Analytics to Analyze Semi-Structured and Unstructured Data rows. 7 Click Merge and Recluster. A message indicates that no additional clusters are found with the fingerprint algorithm. 8 In the Algorithm drop-down list, change the algorithm from Fingerprint to Phonetic. Phonetic algorithms cluster values based on similar word sounds, whereas fingerprint algorithms capture similar word signatures. 9 Select the check boxes in each row for the following words: microstrategy business organizations businessintelligence analytics HyperIntelligence Note: remember, your data might be different because this is live data. You may not see these words. 10 Click Merge and Close. Remove stop words Stop words are commonly used words that provide little information, such as prepositions and conjunctions (after, for, of, but, nor), that do not improve your understanding of sentiment. 1 Select the Tokens column. 2 From the Select Function drop-down list, select Text Search. 34 MicroStrategy

35 MicroStrategy World 2019 Workshop Book 3 A text search area, labeled Tokens for the selected column, opens at the bottom left. Select the Regular Expression check box to enable the use of regex syntax. 4 On your computer, open the stopwords.txt file. Copy the entire text of the file. 5 Return to Desktop and paste the text into the Tokens box. Only the rows in the Tokens column that include these stop words are selected and displayed in the Sample data area. MicroStrategy 35

36 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Note: If no data is displayed in the Sample data area, there is an extra space after the last character of the expression you pasted into the Tokens box. Delete that trailing bit of white space and you will see the stop words appear. 6 Remove the stop words from the data. In the Suggestions area at the top left, click Delete Row Included in Selectors. Remove URLs URLs that are returned by the Twitter API have t.co in the string. In this scenario, the t.co URLs are not helpful so use the Text Search functionality to remove them. 1 To remove the URLs returned by the Twitter API and any other miscellaneous URLs, type http in the Tokens area at the bottom of the window. All rows with URLs appear in the sample data viewer. 36 MicroStrategy

37 MicroStrategy World 2019 Workshop Book 2 In the header of the Tokens area, click the arrow and select Delete Row included in selectors to remove the selected rows. Remove and stem the tokens column 1 Select the Tokens column. 2 From the Select Function drop-down list, choose Text Selector. The text selector area is displayed to the right of the text search area. Both are labeled as Tokens (the selected column). Notice that the text selector area (outlined in red below) provides a count of total distinct tokens with an interactive display of each token and its frequency. MicroStrategy 37

38 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 3 Select the Tokens column again. 4 From the Select Function drop-down list, choose Numeric Selector. The numeric selector displays at the bottom of the window, to the right of the text selector. There should now be three selectors, each of them selecting based on the Tokens column. Verify this by confirming that the column name Tokens is displayed at the upper left of each, as shown below. 5 Depending on your data, the bottom rung of the numeric selector can produce a count or the ratio of numeric to non-numeric characters. Clear the Non-numeric check box, if it is displayed. Select the Numeric check box if it is not already selected. 38 MicroStrategy

39 MicroStrategy World 2019 Workshop Book 6 Remove the numeric rows. From the Select Function drop-down list, select Delete Row Included in Selectors. 7 In the text search area (displayed on the far left), type the simple regular expression [^a-z\d]. This finds all remaining rows that have non-alphanumeric characters. 8 Click the arrow in the middle text selector area and select Cluster And Edit. Because the text selector and text search work together, you are only clustering and editing those rows in the text selector that were found by the text search. 9 No clusters are displayed in the Cluster and Edit window, because of the selected algorithm. In the Algorithm drop-down list, select Phonetic. 10 Select the check box in the Merge column for each row. 11 In the New Cell Value box for each row, type delete. Your Cluster and Edit window should resemble the image below. MicroStrategy 39

40 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 12 Click Merge and Close. 13 In the text search area, displayed on the far left, delete the Regular Expression. All the rows are once again displayed. Notice the change in the items that display in the text selector window area (in the middle). 14 Click the arrow in the text selector area (Tokens) and select Sort by Count. The tokens with the highest frequency are those that you clustered into the delete group and rt, which is the Twitter designation for a retweet. 15 Hover your cursor over delete and click Include. 40 MicroStrategy

41 MicroStrategy World 2019 Workshop Book 16 Hover your cursor over rt and click Include. Both groups are now selected; only those rows are displayed in the data view. 17 In the Suggestions area at the upper left, click Delete Row Included in Selectors. Remove unnecessary punctuation With tweets, very little useful information can be derived from punctuation. However, most punctuation can usually be removed. 1 Select the Tokens column. 2 From the Select Function drop-down list, choose Remove Characters. 3 A new drop-down list is displayed at the top of the window. and then press Enter. Type % and press Enter. This will search for and remove each MicroStrategy 41

42 Use Text Analytics to Analyze Semi-Structured and Unstructured Data punctuation mark, rather than searching In the same way, type each of the following punctuation marks, pressing Enter after each one: : ;,. 4 Click Apply at the top of the window. 5 Now it is time to save our wrangling script. On the top right, we see all the steps we took to wrangle our data. We will save this script so the next time we import similar data, we can automate the process our data prep. Click the Save icon in the script area. Choose the location where you saved the previous script, and save over it with this script. Click Save, then click Close. Finish data import 1 You have completed the tokenization and normalization process, click Apply at the bottom right of the window to apply your data wrangling to the dataset. 2 Click Finish to import the dataset into the dossier. 3 Click Save on the toolbar. Click Browse PC to save your dossier in your My Documents folder MicroStrategy 4 Name the dossier Text Analytics. This file is now a shareable MicroStrategy file (with a.mstr extension). You have completed one of the overarching goals of text analytics: converting unstructured text data into a structured dataset to use for meaningful analysis MicroStrategy, Inc.

43 MicroStrategy World 2019 Workshop Book Exercise 5: Visualize data with a bag-ofwords model Now that each tweet has been segmented into words, you have a bag-of-words model. This is an unordered representation where only the counts of individual words matter. This foundational model allows for a deeper analysis. Create the Token Count derived metric 1 Right-click the Tokens attribute in the Datasets panel and select Duplicate as Metric. 2 Scroll down in the Datasets panel, right-click on the Count (Tokens) derived metric and select Rename. Rename it Token Count. View the data as a grid 2019 MicroStrategy, Inc. 42 MicroStrategy 43

44 Use Text Analytics to Analyze Semi-Structured and Unstructured Data View the data in the default grid format with the Token Count derived metric. This derived metric provides a count of the number of tokens associated with a given Tweet ID. 1 Drag the Tweet ID attribute from the Datasets panel to the Rows drop zone in the Editor panel. Use the Search box in the Datasets panel to search for attributes and metrics in a dataset rather than scrolling through the Datasets panel. 2 Drag the Token Count derived metric to the Metrics drop zone. This allows you to see token frequency when the data is visualized. These text mining methods allow you to highlight the most frequently used keywords in your dataset. View the data as a bar chart 1 Click Insert Visualization on the toolbar to add a new visualization to the dossier. 2 Select Visualization 2. Change its display by clicking Bar Chart from the Visualization Gallery MicroStrategy 3 Drag and drop the Tokens attribute from the Datasets panel to the Vertical drop zone in the Editor panel.

45 MicroStrategy World 2019 Workshop Book 4 Drag and drop the Token Count derived metric to the Horizontal drop zone. 5 In the Editor panel, right-click the Token Count derived metric and select Sort Descending. Filter the bar chart for the most frequent words Create a filter qualification to rank the three most frequently used words in your feed. 1 Click Menu on the bar chart s title bar and select Edit Filter MicroStrategy, Inc. MicroStrategy 45

46 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 2 Click Add New Qualification. 3 From the Based on drop-down list, select the Token Count derived metric. 4 From the Operator drop-down list, select By Rank. From the list of operators below the Operator drop-down list, select Highest. 46 MicroStrategy

47 MicroStrategy World 2019 Workshop Book 5 In the Value box, type 3. 6 Click OK and then click Save. The graph now displays the top three most used tokens. The Filter icon at the top of the visualization s title bar indicates that the visualization is filtered. Remember this visualization is using live data, so your data might not match the image below. MicroStrategy 47

48 Use Text Analytics to Analyze Semi-Structured and Unstructured Data Add a Word Cloud visualization One of the best methods for visualizing word frequency is a word cloud, which sizes words to indicate their frequency in the data. Word clouds highlight popular or trending words in our tweets. 1 On the toolbar, click Insert Visualization. A grid with the heading Visualization 3 is added to the right of the existing grid. 2 Select D3WordCloud from the Custom area of the Visualization Gallery. 3 Move the new visualization above the bar chart visualization MicroStrategy 2019 MicroStrategy, Inc.

49 MicroStrategy World 2019 Workshop Book Filter the word cloud 1 Before you add data to the word cloud visualization, apply a filter to limit the amount of data displayed. Select Visualization 3. 2 Click the Filter icon to display the Filter panel. 3 Drag and drop the Token Count derived metric from the Datasets panel to the Filter panel. 4 By default, a metric filter is displayed as a slider. Change this to a qualification. In the Filter panel, click menu to the right of Token Count, point to Display Style, and select Qualification MicroStrategy, Inc. 48 MicroStrategy 49

50 Use Text Analytics to Analyze Semi-Structured and Unstructured Data 5 Keep the default operator of Greater than. Type 6 in the value box. Add data to the Word Cloud 1 Click Editor to display the Editor panel. 2 Drag Tokens from the Datasets panel to the Attribute drop zone in the Editor panel. 3 Drag the Token Count derived metric to the Metric drop zone. 49 The word cloud should now resemble the following image: 2019 MicroStrategy, Inc. 50 MicroStrategy

51 MicroStrategy World 2019 Workshop Book 4 Click Save on the toolbar to save the dossier. Congratulations You have now completed the Use Text Analytics to Analyze Semi-Structured and Unstructured Data workshop! Over the course of this workshop you have: Imported data from Twitter into MicroStrategy Desktop Used the imported Twitter data to analyze the sentiment scores of Tweets Created a dossier based on the sentiment scores Built a text analytics application Created a bag-of-words model (word cloud image) to visualize the text data For more information on data analysis of semi-structured and unstructured data, MicroStrategy offers the Introduction to Big Data course. For resources and certifications with MicroStrategy Education, visit: MicroStrategy, Inc. 50 MicroStrategy 51

52 1850 Towers Crescent Plaza Tysons Corner, VA Copyright All Rights Reserved. EV-COLL-0969 microstrategy.com