Spotfire Template Automation with Iron Python and Statistical Modeling with TERR. Tom Bernens May 17, :15PM Room

Transcription

1 Spotfire Template Automation with Iron Python and Statistical Modeling with TERR Tom Bernens May 17, :15PM Room

2 Agenda Repetitive analyses Standardizing data formats Automation of gui interactions Other tricks for robust templates Advanced analytics Standardizing inputs Clear documentation

3 Repetitive Analyses When we receive the same type of data over and over, how do we reduce time waste and make our analyses efficient?

4 Repetitive analyses In our business, we collect tons of data! Every part we release has gigabytes of associated data Each piece of data is for a specific purpose GRR, Parametric specifications, characterization If I perform a repeatable study every week, why should I have to start from scratch for every analysis?? Even if we have some preconfigured visuals, why should the end user waste time clicking through the gui?

5 Agenda Repetitive analyses Standardizing data formats Automation of gui interactions Other tricks for robust templates Advanced analytics Standardizing inputs Clear documentation

6 Standardizing on generic data Across the company, data formats for even a specific study can vary wildly. Different hardware, different software, different logging methods Different tests, different devices, different requirements for release How do we go from very specific data sets to very generic data sets? Thanks to Spotfire, we can go between tall/skinny data and short/wide data very easily using the pivot and unpivot transformations. When we standardize templates using tall/skinny data, regardless of where the data originated, it is able to be imported

7 Benefits of Tall/Skinny Data Its generic When measurements are taken, regardless of system, location, time, or any other variable, the data will fit into a standard list of columns Consist of a single column containing the measurement, and the rest of the columns are meta tags describing the measurement Spotfire can handle it very easily Able to split our data apart by meta tags, so now, regardless of the type of test or any other forcing condition, data from all over the company fits into our standard analysis.

8 Great for bulk statistics Test 1 Condition 1 Test 1 Condition 2 Test 2 Condition 1 Test 2 Condition 2 Using Tall/Skinny data Want to see the Cpk (or any statistic) of all of your distributions? A single expression does that math for you

9 Using Tall/Skinny data Great for bulk statistics Easy to segment or combine data in visualizations using hierarchy's Split by one or more meta tags or choose to look at all data across meta tags together. Test 1 Condition 1 Test 1 Condition 2 Test 2 Condition 1 Test 2 Condition 2 Test 1 Condition 1 and 2 Test 2 Condition 1 and 2

10 Agenda Repetitive analyses Standardizing data formats Automation of gui interactions Other tricks for robust templates Advanced analytics Standardizing inputs Clear documentation

11 Automation of gui interactions So we have a dxp file with pre configured visuals, now what? Normally we File -> Replace Data Table -> Add Unpivot Transform -> Add Rows -> Add Unpivot Transform -> Add Rows

12 Automation of gui interactions So we have a dxp file with pre configured visuals, now what? We put Iron Python to work.

13 What is Iron Python? Iron Python is an open-source implementation of the Python programming language which is tightly integrated with the.net framework. Why IronPython It can be used as a fast and expressive scripting language for embedded applications The.NET framework paired with python scripting capabilities makes for an amazing amount of functionality and power.

14 Automation of gui interactions Basically anything you can do manually through the gui can be automated with iron python. Adding Data Tables Replace Data Tables Pivoting/Unpivoting (and all other transforms) Adding Rows/Columns Forcing tables to refresh Setting document properties Trigging other scripts and more

15 Import Script Uses document properties to modify how the script operates. If the data has limits embedded, we will pivot them out into a separate table to make the file size smaller (reduced duplicated meta tag information) If the data format is Tall/Skinny we import it straight, if its Short/Wide we ask the user for their transform/passthrough inputs and perform the pivot function.

16 Import Script Uses document properties to modify how the script operates. Uses standard windows file open dialog to multi-select files Users often time have more than one source of data, we allow them to select multiple and loop through with an AddRows function

17 Import Script Uses document properties to modify how the script operates. Uses standard windows file open dialog to multi-select files Uses a custom windows form to assign custom column names to standard columns We have 5 standard columns, the user matches the column in there data to standard column. All visuals and expressions then reference a property with the user defined name instead of the standard column name

18 Import Script Uses document properties to modify how the script operates. Uses standard windows file open dialog to multi-select files Uses a custom windows form to assign custom column names to standard columns Uses a custom windows form to allow the user to modify the hierarchy that is used to split the data by. Allows the user to select how to split their data. Give the user this control with some default values for those that don t need it.

19 Import Script Document properties Standard windows forms Custom windows forms With all of these tools in place, users are able to get their data into this template very quickly, regardless of file format extra data in the source file the number of files how to split with the hierarchy columns with non standard names

20 Agenda Repetitive analyses Standardizing data formats Automation of gui interactions Other tricks for robust templates Advanced analytics Standardizing inputs Clear documentation

21 Other tricks Not only do we want to make getting data in easy, we want to make using it easy. This means adding things like.. User controlled variables

22 Other tricks Not only do we want to make getting data in easy, we want to make using it easy. This means adding things like.. User controlled variables Approaching the data with a drilldown mentality This means focusing on a small subset of the data at a time. We use the filters, limit by markings, and limit by expressions to achieve this.

23 Other tricks Not only do we want to make getting data in easy, we want to make using it easy. This means adding things like.. User controlled variables Approaching the data with a drilldown mentality Remember, the more control the user has the more ways your template can be used.

24 Summary Repetitive Analyses With standard data formatting, Iron Python can be used to automate gui interactions to create easy to use templates. One time set up of requirements + saving in the library = very easily repeatable analyses. Using document properties can allow users to control how scripts operate, as well as how the template works. This all saves time and energy that can be better spent

25 Advanced Analytics When we need advanced statistical models and ways to manipulated our data, how do we make this robust and easy on the end user?

26 Advanced analytics We often want to look for the root cause of differences in our data sets using advanced analytical techniques Multivariate ANOVA for parametric data A Slew of waveform pre-processers Waveform Outlier Detection for XY data Residual error Threshold masking

27 Introduction to R What is R Programming Language used for statistical data analysis. An open source implementation of the S+ programming language. Created at the university of Auckland, New Zealand by Ross Ihaka and Robert Gentleman Why R R Libraries implement a wide variety of statistical techniques, including linear and non linear modeling, classical statistics tests, time-series analysis, classification, clustering,, and a lot more. Easily extensible through functions and extensions, and the R community has significant active contribution to new packages.

28 Case Study: Multivariate ANOVA. R vs Calculated Columns We want to perform not only 5-15 One-way ANOVA studies. But also combinations of those variables. E.g. Condition 1/2, 1/3, 1/4 1/n 2/3, 2/4, 2/5 2/n. n-1/n In Calculated columns, this is a nightmare. In R, this is a simple for loop to iterate through the unique combinations of effects.

29 Case Study: One-way ANOVA. R vs Calculated Columns Three calculated columns per study. One line of code per study Impossible to scale Very difficult to combine effects Easily scalable, loop-able Easy to combine effects

30 Standardizing inputs When writing a data function, ease of use is one of the top priorities. A mixture of required and not required variables Table/Column handlers for data sets Value handlers for inputs that affect the way the script runs

31 Clear Documentation The other side of this coin is that the function should be well documented for ease of use. What does it do? How does it do it? How do I use it? What does the result look like? We host an internal forum where all functions are documented Also host examples of the script in action in the library for users to access

32 Summary Advanced Analytics R Can be used to easily supplement where Spotfire falls short. Advanced Analytics are easy to achieve in a very high level scripting language. Using TERR, to specify inputs/output is easy to do.