Welcome. Atlanta R Users Group. HPCC Systems Architecture Overview & R Integration Demo

Transcription

1 Welcome Atlanta R Users Group HPCC Systems Architecture Overview & R Integration Arjuna Chala, Architect Integrations, HPCC Systems / LexisNexis Agenda 12:00-12:30pm: 12:30-1:30pm: 1:30-1:50pm: 1:50-2:00pm: Welcome Lunch / Meet & Greet HPCC Systems Architecture Overview & R Integration Demo Q&A / Open Discussion Raffle / Kindle Fire giveaway / Close Twitter event hashtag: #hpccmeetup hpccsystems.com 1

2 Contents -Introducing HPCC -How does LexisNexis use HPCC? -ECL -R and HPCC A match made in Heaven? 2

3 What is HPCC? 3

4 Thor Architecture 4

5 Thor Architecture (contd..) 5

6 Roxie Architecture Distributed Architecture Highly Concurrent Low Latency Highly Scalable Highly Redundant 6

7 HPCC Trivia You have several million records that needs to be cleaned, linked and mined. Which HPCC component will you use? 7

8 To Summarize - Three main HPCC components HPCC Data Refinery (Thor) HPCC Data Delivery Engine (Roxie) Enterprise Control Language (ECL) Massively Parallel Extract Transform and Load (ETL) engine Built from the ground up as a parallel data environment. Leverages inexpensive locally attached storage. Doesn t require a SAN infrastructure. Enables data integration on a scale not previously available: Current LexisNexis person data build process generates 350 Billion intermediate results at peak Suitable for: Massive joins/merges Massive sorts & transformations Programmable using ECL A massively parallel, high throughput, structured query response engine Ultra fast low latency and highly available due to its read-only nature. Allows indices to be built onto data for efficient multi-user retrieval of data Suitable for Volumes of structured queries Full text ranked Boolean search Programmable using ECL An easy to use, declarative data-centric programming language optimized for large-scale data management and query processing Highly efficient; automatically distributes workload across all nodes. Automatic parallelization and synchronization of sequential algorithms for parallel and distributed processing Large library of efficient modules to handle common data manipulation tasks 8

9 How does LN use HPCC? 9

10 Getting Caught in the Act - A LexisNexis Use Case 10

11 Getting Caught in the Act - A LexisNexis Use Case 11

12 Where is John Smith Now? - A LexisNexis Use Case 12

13 Demo Time - SALT 13

14 Insurance Collusion in Louisiana - A (yet another) LN Use Case 14

15 Insurance Collusion in Louisiana - A (yet another) LN Use Case BEFORE AFTER HPCC 15

16 We do have some fun once in a while - A (fun) LN Use Case 16

17 HPCC Trivia Name two attributes that make Roxie a great data delivery engine? 17

18 And Finally.. 12 million background checks a year Big Data Supporting 90 percent of the Fortune 500 companies 99% of all U.S. auto insurance claims Open Source Components 4 Petabytes of Data 30,000 Data Sources 50 billion records Several million records daily 250 million unique identities

19 ECL 19

20 ECL is SQL on Steroids ECL SELECT persons Select * from persons FILTER persons(firstname= Jim ) Select * from persons where firstname= Jim SORT SORT(persons, firstname) Select * from persons order by firstname COUNT COUNT( Person(firstName= TOM )) SQL Select COUNT(*) from Person where firstname= TOM GROUP DEDUP(persons, firstname, ALL) Select * from persons group by firstname AGGREGATE SUM(persons, age) Select SUM(age) from persons Cross Tab TABLE(persons, {state; statecount:= COUNT(GROUP);}, state) Select persons.state, COUNT(*) from persons group by state JOIN JOIN(persons,state,LEFT.state=RIGHT.code) Select * from persons,states where persons.state=states.code 20

21 ECL for ETL Basic Data Structure PersonRec := RECORD STRING50 firstname; STRING50 lastname; UNSIGNED1 age; END; Transformations PersonRec persontransform(personrec person) := TRANSFORM SELF.upperFirstName := UPPER(person.firstName); SELF := person; END; upperpersons := PROJECT(persons, persontransform(left) ); OUTPUT(upperPersons); Functions Used in context of Transformations All Functions PROJECT, ROLLUP, JOIN, ITERATE, NORMALIZE, DENORMALIZE /community/docs/ecl-languagereference/html/built-in-functions-and-actions 21

22 Enterprise Control Language (ECL) Declarative programming language: Describe what needs to be done and not how to do it Powerful: Unlike Java, high level primitives as JOIN, TRANSFORM, PROJECT, SORT, DISTRIBUTE, MAP, etc. are available. Higher level code means fewer programmers & shortens time to delivery Extensible: As new attributes are defined, they become primitives that other programmers can use Implicitly parallel: Parallelism is built into the underlying platform. The programmer needs not be concerned with it Maintainable: A high level programming language, no side effects and attribute encapsulation provide for more succinct, reliable and easier to troubleshoot code Complete: Unlike Pig and Hive, ECL provides for a complete programming paradigm. Homogeneous: One language to express data algorithms across the entire HPCC platform, including data ETL and high speed data delivery. 22

23 Demo Time - ECL 23

24 HPCC Trivia What does ECL stand for? Is ECL meant to be imperative? 24

25 Finally..R and HPCC A match made in Heaven? 25

26 Seen this Before? Data don t make any sense, we will have to resort to statistics 26

27 And the next thing you know 27

28 With HPCC and R you can. Data Sources Analyze, Mine, Model Big Data Processing Business Intelligence Unstructured Data HPCC R ECL Input Data Results Status ECL Results JDBC SQL Results Visualization RDBMS DW Structured Data provide an end to end modeling/analytical solution 28

29 Use the power of HPCC in R 29

30 How did we do it in R? S4 Classes -> Generates ECL code -> Executes on HPCC -> Results back to R 30

31 Q&A Thank You Web: info@hpccsystems.com Contact us: