Pervasive DataRush TM

Transcription

1 Pervasive DataRush TM Parallel Data Analysis with KNIME

2 Company Overview Global Software Company Tens of thousands of users across the globe Americas, EMEA, Asia ~230 employees worldwide Strong Financials $46 million revenue (Trailing 12-month) 40 consecutive quarters of profitability $36 million in the bank NASDAQ:PVSW since 1997 Leader in Data Innovation Cloud-Based and On-Premises Data Integration Data Management Web-based Business-to-Business Data Interchange Highly Parallel Data-Intensive and Analytic Applications 2

3 Complexity The Challenge of Big Data HPC climate modeling seismic analysis fluid dynamics Need to deal with increased data and complexity Enterprise data custom solutions data quality data analytics Internet scale web indexing web search GB Data Size PB 3

4 Multicore SMP Cluster Hadoop Cluster Pervasive DataRush a parallel dataflow platform that eliminates performance bottlenecks in your data-intensive applications Scalable: Performance dynamically scales with increased core counts and increased nodes. High Throughput: Fast, deep analysis of large data sets with no limit on input data size. Analytics and Big Data Application Cost Efficient: Maximum performance from commodity multicore servers, SMP systems and clusters. Easy to Implement: No complex parallel processing issues; visual and API level interfaces. Extensible: Extensible platform so you remain in control of development. DataRush Apps Scale Up and Out 4

5 Pervasive DataRush Architecture High Performance Data-intensive Application KNIME PDR Modules DR DataMatcher DR Recommender DR Profiler User-defined Modules Data preparation Data analytics DR Core Data Prep Lib DR Core Analytics Lib User-defined Libraries DataRush SDK Dynamic Processing Graph Large volumes of data PDR Parallel Dataflow Engine JVM: Java, Python, JRuby, SCALA Quality data Actionable analytics 5

6 DataRush & KNIME Integration Desktop plug-in for DataRush usage Nodes for data preparation and manipulation Base set of parallelized data mining functionality Highly efficient & parallelized data staging Parallel execution extension SDK plug-in for DataRush node development Create your own DataRush based nodes Access to full DataRush API s Wizard for creating DataRush based nodes 6

7 Normal Execution 7

8 Normal Execution DataRush Engine spawns 8

9 Normal Execution DataRush Engine spawns 9

10 Normal Execution spawns DataRush Engine 10

11 Normal Execution - Complete 11

12 Parallel DataRush Executor Capabilities Supports parallel execution of DataRush based nodes without intermediate staging Automatically splits workflows into executable graphs at staging boundaries Executes non-datarush nodes including metanodes, for loops and branches Usable within desktop, command line and server environments 12

13 Parallel Execution 13

14 Parallel Execution DataRush Engine 14

15 Parallel Execution DataRush Engine 15

16 Parallel Execution - Complete 16

17 Parallel Execution Details Parse Replace Aggregate Format Parse Parse Replace Replace Aggregate Aggregate Format Format Write Parse Replace Aggregate Format 17

18 Demo

19 Vision for Levels of Usage Level 0 No code changes on your part Install DataRush plug-in and most nodes will see a performance benefit Level 1 Some code changes required Utilize DataRush to access parallelized data staging capability bypassing BDT API Level 2 Utilize DataRush SDK to build nodes using the full parallelized flow capability of DataRush Available today 19

20 Demo

21 DataRush Benefits High Throughput Process data quickly and efficiently Accomplish complex processing in a single pass Scalable Takes advantage of multicore processors Runs faster as more cores are added Scales with the amount of data Easy to use and extend Dataflow abstraction hides parallelism details SDK to ease development

22 Summary Scale performance on commodity multicore systems Massive performance exists on a single server Core counts growing with Moore s Law Scale up and scale out Economical, environmental, and manageable Scale to big data Handle diverse, complex, massive data sets Scale development Easy for existing team to implement parallel applications Extensible platform keeps you in control Simplify how you develop Big Data applications 22

23 Questions?