Analyst for Kx. Product Overview

Size: px

Start display at page:

Download "Analyst for Kx. Product Overview"

Rose Kathlyn Wiggins
6 years ago
Views:

1 Product Overview

` Analyst for Kx Real-Time Data Transformation, Exploration and Discovery Suite Analyst For Kx provides users with a complete real-time data transformation, exploration and discovery workflow.

2 ` Analyst for Kx Real-Time Data Transformation, Exploration and Discovery Suite Analyst For Kx provides users with a complete real-time data transformation, exploration and discovery workflow. Using an intuitive point-and-click interface, the typical analyst can import, transform, filter, and visualize massive datasets without programming. If the user is comfortable with a small amount of scripting, they can use the Spreadsheet to perform simple or complex analytical computations on their massive datasets in real-time. The Analyst For Kx product suite comprises the following components: Workspace Importer Transformer Visualizer Spreadsheet Exporter Version control and share entities and datasets Import datasets into the Workspace Groom, transform, filter and query datasets Visualize massive datasets in real-time or near-real-time Perform analysis using a simplified SQL query and scripting language Export datasets into other systems. 1

3 Solution Overview Analyst for Kx provides users with a complete real-time data transformation, exploration and discovery workflow. Using an intuitive point-and-click interface, the typical analyst can import, transform, filter, and visualize massive datasets without programming. If the user is comfortable with a small amount of scripting, they can use the Spreadsheet to perform simple or complex analytical computations on their massive datasets in real-time. The Analyst For Kx product suite comprises the following components: Workspace Importer Transformer Visualizer Spreadsheet Exporter Version control and share entities and datasets Import datasets into the Workspace Groom, transform, filter and query datasets Visualize massive datasets in real-time or near-real-time Perform analysis using a simplified SQL query and scripting language Export datasets into other systems The suite is designed to augment the analyst workflow illustrated below The rest of this document summarizes the function of each tool in the suite. 2

Workspace The Workspace lets the analyst store, version control and share their analysis with others.

The tree shows the analyst s local repository containing a project with two CSV files.

From the tools menu, the analyst can access the Importer, Transformer and Visualizer.

4 Workspace The Workspace lets the analyst store, version control and share their analysis with others. In the example below, we can see the Workspace with a tree structure of artefacts on the left and a work area on the right. The tree shows the analyst s local repository containing a project with two CSV files. The analyst can also look at the shared repository to see the public artefacts in the system and can publish artefacts to the shared repository. From the tools menu, the analyst can access the Importer, Transformer and Visualizer. Importer The Importer allows the analyst to import CSV, JSON, ODBC and Kx tables into their workspace using a wizardstyle interface. Users can select a file from either their local machine or the server. The screen shot below shows the second step in the import wizard where the analyst can change column names and adjust column types if they do not like the automatic column name and type detection performed by the importer. In addition, they can sanitize the column names to trim blanks and remove keywords. When finished, the analyst has the option of immediately importing the file into the workspace, the Transformer for additional manipulation, into the Visualizer for exploration or saving the file to disk in CSV, JSON, ODBC or Kx formats. 3

Transformer The Transformer is a powerful extract, transform and load (ETL) tool that allows the analyst to perform complex data grooming operations on your data using a point-and-click interface.

of the columns from short integers to long integers. The top pane provides the user with a visual map of the transformation.

5 Transformer The Transformer is a powerful extract, transform and load (ETL) tool that allows the analyst to perform complex data grooming operations on your data using a point-and-click interface. In the example below, the user has loaded two tables into the transformer, created a new table by performing a left join on the two tables, and modified the join output by changing the types of two of the columns from short integers to long integers. The top pane provides the user with a visual map of the transformation. The lower left pane provides the user with a list of actions have been applied to the currently selected table. The lower right pane provides the user with a sample of the updated output so that they can verify that the actions have been applied correctly. Users apply actions to the currently selected dataset using the main menu or a context menu. 4

The following table describes the actions that are supported in the Transformer: Action Update Description Allows the user to update

For example, here the analyst uses the point-and-click interface to add an action that sets the duration column to zero if the source

Multiple columns can be updated at the same time and complex nested and/or qualifications can be expressed in the interface.

6 The following table describes the actions that are supported in the Transformer: Action Update Description Allows the user to update a column s data value based on simple or complex and/or query operations. For example, here the analyst uses the point-and-click interface to add an action that sets the duration column to zero if the source column is equal to 300. Multiple columns can be updated at the same time and complex nested and/or qualifications can be expressed in the interface. Filter Allows the user to filter the dataset based on simple or complex and/or operations using a point-and-click interface. Add columns Allows the user to add new columns. The value of the column can be any valid q statement (e.g. a custom function, a column name, a variety of standard aggregation operations) 5

Transform type Allows the user to change the type of any column to any standard q type. The context menu also contains some commonly used shortcuts.

Transform cells Allows the user to perform data filling operations: Transform column Allows the user to perform a wide range of column operations.

7 Transform type Allows the user to change the type of any column to any standard q type. The context menu also contains some commonly used shortcuts. The Parse Numeric feature allows users to parse strings and symbols into number formats. Transform cells Allows the user to perform data filling operations: Transform column Allows the user to perform a wide range of column operations. Column attributes (e.g. sorted, keyed) can dramatically improve query performance. Expansion of column dictionaries and splitting columns on value separators are particularly useful for JSON, URL and Log File data. Transform text Allows the user to perform basic text operations: 6

Joins Allows the user to create left, inner, plus, union, append, insert and zip (null padding or join on index) joins. For more information about joins, please consult the Kx reference manual.

You can also use any custom function you wish to create. It is similar to a stored procedure in SQL but far more powerful.

8 Joins Allows the user to create left, inner, plus, union, append, insert and zip (null padding or join on index) joins. For more information about joins, please consult the Kx reference manual. Canned and custom functions The Transformer allows you to use common statistical values (e.g. count, average, maximum, minimum, distinct, standard deviation, first, last) in the value field. You can also use any custom function you wish to create. It is similar to a stored procedure in SQL but far more powerful. In the example below, the user has created a custom function that takes two column values and performs a mathematical computation on the columns and returns the result. The user then uses the function in the value field and passing two other data columns to the function: 7

It also adds a transformation node. The user right-clicks on columns in the transform table and adds two actions. The first action changes the souce column from short to long integer.

9 Transformer Workflow The following steps describe how a user might use the transformer in a typical analysis situation: 1. The user adds a column to the transform area by selecting Table>Open from the main menu and completing the wizard. 2. The transformer updates to show the added table. It also adds a transformation node. The user right-clicks on columns in the transform table and adds two actions. The first action changes the souce column from short to long integer. The second action changes the dest column from short to long integer. The actions appear in the lower left pane and the outcome of the action is displayed on the data in the lower right sample pane. The user can delete any action from the list, turn the action on or off to see the effect of the action, or reorder the actions by dragging and dropping the actions in the list. Any table in the transformation graph can have an action list associated with it. 8

10 3. The user now adds a second table in the same way. 4. Now the user joins the two tables by selecting the first table and choosing Join Table from the context menu. 5. The Join Dialog appears and the user selects the kind of join they wish, the columns they wish to join, and the columns they wish to see in the resulting output table. In this case, the user elects to perform a left join against the tower column. 9

6. The result of the join is shown in the transformation table below. 7.

For example, they may elect to perform one set of filtering operations on the dataset and

11 6. The result of the join is shown in the transformation table below. 7. The user can add different views of the output to aid in their exploration. For example, they may elect to perform one set of filtering operations on the dataset and another set of filtering operations on the dataset to obtain to distinct outputs as shown below: 8. Once the user is satisfied with their explorations, the user can export any aspect of the transformation graph to memory or disk. In the example below, the user chooses to export the unfiltered join table. 9. The Exporter wizard appears and the user chooses to export the data as a Kx in-memory table. Users can export data into CSV, JSON, ODBC or Kx (memory, serial, splayed, partitioned) formats. 10

10. They can also save the transformation so that they can work on it later or share it with their colleagues.

For example, supposed that the user received a call database every day and needed to perform this same transformation operation over and over again.

When the user looks at the Workspace, they can see CSV files, the custom function and the transformation in the.myproject module.

12 10. They can also save the transformation so that they can work on it later or share it with their colleagues. In addition, the saved transformation can now be executed as a function in a program with other data tables that share the same schema. For example, supposed that the user received a call database every day and needed to perform this same transformation operation over and over again. They could build this transformation once and then execute it as a program against their daily files. 11. When the user looks at the Workspace, they can see CSV files, the custom function and the transformation in the.myproject module. Each of the artefacts is decorated with a plus icon to indicate that these artefacts are newly created and not yet published to the shared repository. At this point, only the current user can see these artefacts. The user publishes artefacts to the shared repository by right-clicking on the module and pushing it into the shared repository. Similarly, they can pull other modules from the shared repository or update to a specific version of a module in either the local or shared repository. 12. At this point, the user can use the Visualizer to explore their data in real-time. 11

This 200 x 200 binned histogram visualization (top) takes between 4 and 7 seconds depending on network latency and server load (using 2 cores of an Amazon

13 Visualizer The Visualizer is designed to allow users to render huge datasets in a variety of visual formats. In the example below, the user is visualizing 200,456,232 records. This 200 x 200 binned histogram visualization (top) takes between 4 and 7 seconds depending on network latency and server load (using 2 cores of an Amazon ec2 r3.4xlarge instance). Furthermore, rendering is linear so a 2000 x 2000 binned heatmap (bottom) takes the same amount of time. This approach makes it possible to treat big fast data as if it were small data by allowing the user to tumble, turn and explore the data in effectively real-time. 12

14 Users can dynamically switch between chart types on-the-fly using a point-and-click interface: Users can manipulate a wide range of chart properties or attributes on-the-fly using a point-and-click interface: 13

Users can manipulate the following chart properties or attributes: Attribute Chart Selection Column Selection Column Binning Column Grouping Alpha Aggregation Fill Aggregation Color Aggregation Drill

15 Users can manipulate the following chart properties or attributes: Attribute Chart Selection Column Selection Column Binning Column Grouping Alpha Aggregation Fill Aggregation Color Aggregation Drill Down Drill Across Save Version Control Description The following table types are supported: Table, Histogram, Horizontal Histogram with sub groupings Heatmap Box, Scatter, Quantile Bar, Line, Horizontal Bar Path Geographic map mercador longitude and latitude, Directed graph Google map Any column in the dataset can be map to x and y coordinates Any axis can be bin in any number of increments Depending on the graph type, columns can be grouped in series or sub-grouped to provide pivot table behavior The following aggregations are supported: Count,Average, Max, Min Distinct, First, Standard Deviation The following aggregations are supported: Count, Average, Max, Min Distinct, First, Standard Deviation The following aggregations are supported: Count, Average, Max, Min Distinct, First, Standard Deviation Users can drill and up down into any region on a chart. Users can zoom in and out of a directed graph. Users can drill across at any level within a chart and change the chart attributes as they drill across. For example, the user drills down on longitude and latitude columns, then decides to drill across based on source and destination columns at the current depth. Charts remember their history so users step backwards through the drill path. Users can save visualizations. Users can version control the visualization. Users can switch to different versions of the visualization. Users can push visualizations to the shared repository. Users can pull visualizations from the shared repository. Note: Dynamic qsql Queries While the Visualizer can be used with the Transformer to perform point-and-click queries, a query line is provided at the top of the Visualizer so that sophisticated users can enter text queries directly into the Visualizer. In the example below, the user selects the distinct sources grouped by destination from the table where the destinations are 0, 1, 5, 306, or

16 Sample Charts Table Chart Example with paging Histogram 15

17 Horizontal Histogram with sub-grouping Heatmap 16

18 Box Scatter 17

19 Quantile Bar 18

20 Horizontal Bar Line 19

21 Path Geographic map mercador longitude and latitude 20

22 Directed graph Google map 21

The spreadsheet can be used in conjunction with the Importer, Transformer and Visualizer.

23 Spreadsheet More sophisticated users can elect to use the Spreadsheet to perform simple or complex data analysis. The sheet is similar to Excel with many of the same features but it allows users to load massive datasets without performance limitations. The spreadsheet can be used in conjunction with the Importer, Transformer and Visualizer. Like these tools, the Spreadsheet is version managed and can be shared across users depending on access permissions. In the example below, a table containing about 2.5 million records is loaded into the sheet. The user then performs standard SQL query operations in real-time against the dataset. In the first example, the user is selecting all the records where the source is less than 5. In the second example, the user is returning a single value that is the absolute sum of the duration of all the records in B2. 22

24 In addition to performing calculations, the user can access the Real Time Visualizer at any time by right-clicking a cell and inspecting the cell contents. The Visualizer can view any kind of data including standard values (e.g. integers, strings, floats) as well as lists, dictionaries, tables and key tables. 23

Finally for development teams, the Spreadsheet can be programmed to create front-end user interfaces for novice users. In this example, a combo box has been added to a sheet using JavaScript.

25 Finally for development teams, the Spreadsheet can be programmed to create front-end user interfaces for novice users. In this example, a combo box has been added to a sheet using JavaScript. In that way, more sophisticated users can turn complex operations into simple point-and-click interfaces using a small set of helper functions. Here is the corresponding JavaScript script to create the combo box: 24

26 For more information For more information about this and other our other products, please visit 25

27 EMEA Head Office 3 Canal Quay, Newry, BT35 6BP N. Ireland Tel: +44 (0) Belfast Gloucester Street, BT1 4LS N. Ireland Tel:+44 (0) Dublin Fleming Court, D04 N4X9 Rep. of Ireland Tel: +353 (0) London Cannon Green Building, 1 Suffolk Lane, EC4R 0AY United Kingdom Tel:+44 (0) Americas New York 45 Broadway, New York, NY USA Tel:+1 (212) Toronto 1599 Hurontario Street Mississauga, On, L5G 4S1 Canada Tel: +1 (905) Ottawa 300 Terry Fox Drive, Kanata, On, K2K 0E3 Canada Tel: + 1 (613) Palo Alto # Bryant Street, CA USA Tel: +1 (650) APAC Sydney 22 Pitt Street, Sydney, NSW 2000 Australia Tel: +61 (0) Singapore 55 Market Street, Singapore Tel: Hong Kong Two Exchange Square, 8 Connaught Place, Central Tel: Tokyo Sanno Park Tower, Nagata-cho, Chiyoda-ku, Japan Tel:+81 (0) Kx and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

Kx Analyst. Value Proposition: Features and Benefits. Kx.com

Kx Analyst. Value Proposition: Features and Benefits. Kx.com Kx Analyst Value Proposition: Features and Benefits Kx.com Topics Covered What Is KX Analyst? Who Uses KX Analyst? Product At A Glance Use Cases Value Propositions Highlighted Features 2 What is KX Analyst?