Scripting without Scripts: A User-Friendly Integration of R, Python, Matlab and Groovy into KNIME

Transcription

1 Scripting without Scripts: A User-Friendly Integration of R, Python, Matlab and Groovy into KNIME Felix Meyenhofer Technology Development Studio 3. March th KNIME Users Group Meeting and Workshop 1

2 Outline 2 2

3 Outline 2 2

4 Our situation 3 3

5 The actors I m a scientist. I use Excel because I rely on a proper UI, but it does not scale, does not work well for all types of experiments, etc.. So I often need to go to see my data analyst I m a data analyst. I can script in R, Python, Matlab and a little Java. I hate user interfaces! 4 4

6 Are data analysts lazy people? A (typical) research project Experiment Lots of data Fancy data mining scripts A little Excel Great results Paper 5 5

7 Are data analysts lazy people? A (typical) research project Experiment Lots of data Fancy data mining scripts A little Excel Great results Answer: No, but staff-ratio = 8:1 Paper 5 5

8 Can fix the problem? + It is able to handle big data sets + Has ready to use data interfaces (txt, xls, sql...) + It s modular architecture allows customization - Native KNIME visualisation are insufficient (for our purpose) - It lacks specific methods and tool - node developement requires java-geeks and takes time, where the data analysis requirements evolve quickly along with the experiments In contrast: Scripting solutions can be quickly developed but are not end-user ready Scripting allows rapid prototyping of methods and tools 6 6

9 Idea What if data analysts could write their scripts as usual, AND users could somehow access these within a well designed framework using a graphical interface? + = 7 7

10 The basic components Java KNIME Scripting Framework Server + API Module 8 8

11 Script template system XML template User interface Script Visne et al., RGG: A general GUI Framework for R scripts, 2009, Bioinformatics, doi: /

12 Scripting node state diagram Empty Script Scripting Create Node Not rgg Execute on execute Script Unlink Reattach/ Convert to template Template Create UI Configure Discards all script customizations Edit Template Rebuild UI X 10

13 Reduction to the necessairy internal internal 10 11

14 Template management Local or remote template repositorie Plain-text template definition files Hierarchically organized Previews for visualization templates 11 12

15 The (new) situation Facilitated knowledge propagation within and among research groups Facilitated knowledge (work) preservation with the template system Applicable form different levels of expertise Users Use Knime and templates Be happy! Advanced Users Customize templates in-place Use flow variables and loops Data Analysts Write scripts to fill Knime-gaps Evolve scripts into new templates Developers Provide Knime-nodes for most popular templates 12 13

16 The (new) situation Facilitated knowledge propagation within and among research groups Facilitated knowledge (work) preservation with the template system Applicable form different levels of expertise Users Use Knime and templates Be happy! Advanced Users Customize templates in-place Use flow variables and loops Data Analysts Write scripts to fill Knime-gaps Evolve scripts into new templates Developers Provide Knime-nodes for most popular templates 12 13

17 Scripting nodes vs. conventional nodes Less integrated than native nodes Real nodes are likely to be (much) faster Real nodes scale better/work with huge data-sets Real nodes can be updated by updating Knime Real node-development requires 10x more resources 13 14

18 From templates to nodes + = Node Templates become updateable Rapid R prototyping Trivial deployment (1node/min) into end-user ready nodes 14 15

19 From templates to nodes + = Node Templates become updateable Rapid R prototyping Trivial deployment (1node/min) into end-user ready nodes 14 15

20 Scripting language support R Rserve as backend 15 16

21 Scripting language support R MATLAB Rserve as backend mpicbg-matlab web-client (needs a floating license!) 15 16

22 Scripting language support R MATLAB Python Rserve as backend mpicbg-matlab web-client (needs a floating license!) mpicbg-python backend 15 16

23 Scripting language support R MATLAB Python Groovy Rserve as backend mpicbg-matlab web-client (needs a floating license!) mpicbg-python backend Using Knime API but in a dynamically complied environment 15 16

24 Scripting language support R MATLAB Python Groovy Rserve as backend mpicbg-matlab web-client (needs a floating license!) mpicbg-python backend Using Knime API but in a dynamically complied environment server infrastructure makes it easier for maintenance. (power users can use a local instance) 15 16

25 Scripting language support R MATLAB Python Groovy Rserve as backend mpicbg-matlab web-client (needs a floating license!) mpicbg-python backend Using Knime API but in a dynamically complied environment What else? All that is necessary is (1) a table conversion mechanism, (2) the possibility to invoke a script with the converted table as argument, and (3) a way to convert the results back into a table 15 16

26 Feature overview R MATLAB Python Groovy features STATISTICS Visualization Handy data-types Statistics IMAGE PROCESSING Visualization Easy Java interface Documentation BIOINFORMATICS Statistics (Image Processing) Visualization (Matplotlib) Java prototyping regexp... Exponential growth in packages Many people know it (#29) * #4 progamming language * )-: memory management rather slow without grid computing toolbox interface syntax (just joking) *

27 Knime is focused on table transformations Some problems do not fit into this scheme Solution: nodes 17 18

28 The MATLAB integration trio 18 19

29 The MATLAB integration trio 18 19

30 The MATLAB integration trio 18 19

31 The MATLAB integration trio 18 19

32 The MATLAB integration trio 18 19

33 The MATLAB integration trio 18 19

34 The R integration trio 19 20

35 The R integration trio 19 20

36 The R integration trio 19 20

37 The R integration trio 19 20

38 The R integration trio 19 20

39 The R integration trio 19 20

40 The Python integration trio 20 21

41 The Python integration trio 20 21

42 Creating a Python plot template 20 21

43 Creating a Python plot template 20 21

44 Creating a Python plot template 20 21

45 Creating a Python plot template 20 21

46 Creating a Python plot template 20 21

47 Creating a Python plot template 20 21

48 Creating a Python plot template 20 21

49 Creating a Python plot template 20 21

50 Creating a Python plot template 20 21

51 Creating a Python plot template 21 22

52 Creating a Python plot template 21 22

53 Creating a Python plot template 21 22

54 Creating a Python plot template 21 22

55 Creating a Python plot template 21 22

56 Creating a Python plot template 21 22

57 Question to the audience: Is this KNIME or R? 22 23

58 Question to the audience: Is this KNIME or R? 95% 22 23

59 Our experience so far: I love templates, because they provide me the exact tools I need. And my data analyst can create them almost instantaneously. I love templates, because they can be used to quickly make custom solutions accessible for my scientists

60 Aggregation Additional features of our scripting integrations: Better editor (undo, redo) Preserve column names Better attribute name insertion UI templates + centralized user template repositories Worker engine can be run locally or on a centralized server Dynamic repainting of plots OpenIn* nodes for rapid prototyping Faster table conversion (~2x for R) Flow-variable support in scripts nodes to process arbitrary data structures Additional scripting languages: Python Matlab 24 25

61 Acknowledgements Holger Brandl (Software developer for Bioinformatics) Tom Haux (Software developer SWENG) Martin Stöter (Scientist TDS) Michael Berthold and the entire KNIME team 25 26