Aster. Now. Future. Why.

Transcription

1 Aster. Now. Future. Why. Michael McIntire CTO Teradata Labs, Aster #TDPARTNERS16 GEORGIA WORLD CONGRESS CENTER

2 Who is that McIntire guy anyway Extreme Scale MPP platforms & complex data systems The Seven year stretch Seven years: Geographic Information Systems Seven years: Teradata in the 90 s DB Architect Seven years: independent EDW consultant Seven years: ebay EDW Chief Architect And some between time... Yahoo, Sears, Prime, EDS...

3 Roles Market

4 Role One: Ideas Business Process Algorithm and Knowledge Finding Business Process Growth Objective Discover and Prove New: Classic and Citizen Data Scientists REVIEW DESIGN PLAN BUILD TEST DRIVE Speed of Ideation Dynamic Connectivity Broad analytic capability Scalable performance EVALUATE

5 Role Two: Operationalization IT Process Given a Known Hypothesis IT - Fitting things together Generalized Architecture Secure, Repeatable, predictable Platform for Production IT Process Control for Cost Architecture Adherence Enterprise Features and Resiliency Fed by the Business process Production Platform

6 Platform view of the market Expanding Compute Engines Narrower Focus More capable point solutions Diverging Storage and Compute APIs are the Enabler Consolidating Storage Engines Rise of Good enough Compute Storage Former Platforms as Engines... Pluggable Infrastructures CEPH Local

7 Ecosystem is the Platform Connecting Multiple Platforms at Runtime Aster Compute within Hadoop Ecosystem First Class Citizen YARN Resource mgmt Native Read/Write Storage MPP Interoperability with External Engines Native Aster and Spark integration Teradata + Aster + Hadoop Accellerate Business Decision making with Platform Interoperability

8 Engine Workflow Integration RIGHT PLAY WORK & PLAY WORK What kind of play? Play Hard or Money? Creative money? FACEBOOK GAMING EXTREME JUST NOTES NO YES Why not get Snake action game First Shooter Be an Actor What kind of Work? An Opad? FALSE BROKEN Snake action game NO YES NO YES RIGHT NOTEPAD Play Hard or Money? Play Hard or Money? Inputs FALSE Getting A success NOTEPAD NO RIGHT Outcome YES Get a Success RIGHT FALSE FALSE NOTEPAD NO NO Get a Succes s? YES YES Every Analytic Engine will have one Just like visualization First Generation: AppCenter Workflows will Mix Paradigms Set processing + Procedural So What s New: Command Language Implementation Server Side implementation Visual Tools Layered on Commands Data Cooking tools already there Tech issue: Exposing Logic inside a set statement across proc bounds. Predicate Search for example Optimization across loop/branch constructs

9 Aster Next

10 Objective - Aster in the Cloud Aster 6.2x on Appliance Aster on Hadoop (AsterX 7.0) Compute Aster 6.20 on AWS Aster 7.x on Cloud Managed, Public Not likely to see: Aster on Hadoop on the Cloud CEPH Storage Local

11 AsterX Evolution Aster Execution Engine Aster is a Compute Engine Spill to disk temp storage only Compute Non-Persistant Storage Access via Connectors QueryGrid2 CEPH Storage Local

12 6.20 Worker Aster Managed Storage Worker Node vworker ASTER - vworker Many ASTER per - vworker Node Cluster Services 1 per node Map Reduce Engine Graph Engine User Edge Node Queen Exec Relational Engine Aster Aster Compression Replication Aster Aster Local Storage Node Local Storage OS Managed

13 6.50 Worker 100% Storage Hadoop Worker Node vworker m/node ASTER - vworker ASTER - vworker Relational Engine Cluster Services 1 per node Map Reduce Engine Cluster Wide Storage Interface Graph Engine Hado YARN op Name Node Aste r User Edge Node Queen Exec Distributed File System ( on HDP, CDH) Compression Replication Security Management Cluster Cluster Wide Storage: :/aster/vworkerx*y

14 AsterX 7.0 Worker Local Temp Hadoop Worker Node vworker m/node ASTER - vworker ASTER - vworker Relational Engine Cluster Services 1 per node Map Reduce Engine Graph Engine Hado YARN op Name Node Aste r User Edge Node Queen Exec NODE LOCAL Storage: /aster/vworkerx*y Cluster Distributed Persistant Data System Hive, Teradata,

15 Aster 7.x Architecture

16 AsterX 7.0 Cluster Architecture Internally: Daemon based implementation Always on - not per job instantiation vworker Deployment: Cluster Subset vworker count is Static per Instance vworkers can be moved Expand / contract Hadoop Cluster without Aster intervention Architected as a SUBSET User Edge Node App B Queen User Edge Node App A Queen Head Nodes Hadoop Services Name Node Queen edge node (required) Security and Connectivity (++ eliminate bridge) Aster A Aster B Map/ Reduce Libraries on all nodes For Simplicity and Latency reasons Aster B Hive Aster A Worker Nodes

17 YARN Managed Resource Full Hadoop Services Integration Injects Third Party Management Reverse order Worker setup/teardown Yarn Managed Edge Node User ASTER Queen Hadoop Services Ambari Aster Yarn Server Yarn Client 2 1 Zookeeper YARN Aster Cluster still manages State Consul implementation YARN Managed Worker Node 4 ASTER vworker 3 Aster

18 AsterX 7.0 Consul State Management Consul: State management, configurations Simple, always on key-value store Similar to ZooKeeper (Dir/Key structure) Consul User Queen Aster 7.0 use: Common, resilient store port mapping Future use dynamic mapping of ports Dynamic worker movement Consul is required AX7.0 is a Private Implementation Future use of existing Consul possible If not available Aster will not come up Aster Temp Aster Temp

19 AsterX 7.0 Cluster Configuration Subset of nodes: explicitly or system decides Exact # nodes will fit node capacity i.e if the nodes are powerful there will be fewer nodes used Alternate maxusage yarn parm for temp/io heavy apps Equivalent of Prepared state, still needs activate Port Configuration startup time: port conflicts can be resolved Re-address when new Cluster SW is installed User Queen No add/remove node functionality Stand up another cluster Point to the data No data migration... Aster Temp Aster Temp

20 AsterX 7.0 Cluster Startup Install, startup - separate steps Install libraries, basic directories Startup plumbs all connections Setup vworkers Connect Queen and Workers Shutdown - cleans up the workers Temp data removed Reuse temp : Future Optimization Case All via Aster Yarn Client Commands Equivalent to Aster Activate Consul Aster Temp User Queen Aster Temp

21 AsterX 7.0 Worker Local Temp Only Persistence in Hive, hcat Access: Connectors + QueryGrid Read at script Start / Write at script End Objects managed by user Same semantics as a Database Persist for Duration of Cluster No Replication & Compression Redistribution remains Cluster Hado YARN op Name Node Hadoop Node Hadoop Node Hadoop Aster Connectors Node Query Grid vworker m/node ASTER - vworker ASTER SQL - vworker M/R API Engine Engine NODE LOCAL Storage User Edge Node Quee n Exec Graph Engine

22 AsterX Local Storage

23 Advanced Analytics Enabled by SQL (for Data/Business Analysts) Once you know how to use on Aster SQL command you have learned how to use them all! CREATE TABLE complaints_nb_model (PARTITION KEY(token)) AS SELECT token, SUM(crash) AS crash, SUM(no_crash) AS no_crash FROM NaiveBayesText ( ON complaints TEXT_COLUMN ('text_data') CATEGORY_COLUMN ('category') CATEGORIES ('crash', 'no_crash') ) GROUP BY token; ANSI SQL Statement SQL MR Statement Data Source SQL-MR Predicates

24 AsterX 7.0 Storage Examples: Analytic Temp Tables and Hive Perm Tables Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive Table Web log Perm Storage Local Spill to disk AX AX AX SQL Temp Temp Queen TEMP Storage

25 AsterX Storage Before AX is running Hive tables: Hive_t1, Hive_t2, Hive_t3 Flat Files: weblog.txt Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive T1 Hive T2 Hive T3 Web log Perm Storage Local Spill to disk AX AX AX SQL Temp Temp Queen TEMP Storage

26 AsterX Storage CTAS analytic: Aster_analytic_t1 CTAS Temp: Aster_session_temp_t2 Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive T1 Hive T2 Hive T3 Web log Perm Storage Local Spill to disk AX AX AX SQL Temp Temp T1 T2 TEMP Storage Queen Uptime Lifetime Session Lifetime

27 AsterX Storage SQL query phase temp tables Temp_phase_1 (the real name would be like _tmp_ ) Temp_phase_2 Query_output Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive T1 Hive T2 Hive T3 Web log Perm Storage Local Spill to disk AX AX AX SQL Temp Temp Queen T1 T2 TP1 TP1 QO Query Lifetime TEMP Storage

28 AsterX Storage CTAS To Hive Alan_dailyreport_06_24_2015 Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive T1.. Hive Web Alan T3 log DR Perm Storage Local Spill to disk AX AX AX SQL Temp Temp T1 T2 TEMP Storage Queen

29 AsterX Storage After Aster shutdown Foreign Server Read/Write w SQL-H Distributed Storage: Hive, hcat Hive T1.. Hive Web Alan T3 log DR Perm Storage Local Spill to disk AX AX AX SQL Temp Temp Queen TEMP Storage

30 AsterX Failure & Recovery

31 Metadata persistance Admin DDL Checkpoint Saves checkpoint file to disk Manually done via ncli command Restart causes checkpoint to be replayed Checkpoint files are valid on any AsterX instance* Check pointed Users, Roles, Databases, Schemas foreign server definitions packaged analytics models and functions grant privileges on above Not Check pointed Tables, views, constraints, indexes R scripts installed on the server side user-installed files and SQL/MR functions user scripts for vacuum or daily jobs

32 AsterX 7.0 Failure Recovery Node Failure = loss of analytic tables Edge Node Queen User Worker Node AND/or vworker System will allocate new node Conversation with YARN Move vworkers/node Come to prepared state Activate automatically Before After Restart Aster Aster Temp Temp Temp Edge Node Queen User ALL TEMP Data is LOST. Vworker is treated as node failure Aster Aster Temp Temp Temp

33 AsterX 7.0 Failure Recovery Queen Fails Recovery is Repair DDL Gen Unwind User Edge Node App B Queen User Edge Node App A Queen Head Nodes Hadoop Services Name Node If unrecoverable Delete cluster cluster create Aster A Aster A Map/ Reduce Aster B Hive Aster A Worker Nodes

34 AsterX 7.0 Failure Recovery Other issues Same behaviors, different impact User Edge Node App B Queen User Edge Node App A Queen Head Nodes Hadoop Services Name Node DDL Gen SQL Script of the dictionary. Aster A State is lost Temp data will be deleted on restart Aster A Map/ Reduce Aster B Hive Aster A Worker Nodes

35 AsterX 7.0 Expansion New Instance. Got that? User Edge Node User Edge Node Head Nodes Hadoop Services Create New Aster Instance Setup Foreign Server Constructs App B Queen App A Queen Name Node Go Aster A Reference Existing Persistent Data Aster A Map/ Reduce Aster B Hive Aster A Worker Nodes

36 AsterX 7.x Configuration Options Many, many more options User User Head Nodes Single cluster per workload Or... Xmas sized Cluster... Monthly Term licensing??? Edge Node App B Queen Edge Node App A Queen Hadoop Services Name Node Internal Chargeback? Aster A LOB specific Aster Instance Delegation of adminstration... Simplified CapEx / OpEx administration Aster B Hive Aster A Aster A Map/ Reduce Worker Nodes

37 Aster Persistent Storage & Access - Query Grid Two

38 Aster 7.10 QueryGrid Two - Next Gen High speed TD, Presto, Hadoop connectivity Cluster to Cluster connectivity Point to Point model not hub (Kafka is a Hub) Common Framework included in each product Communications, State, Error Management, Data Conversion Network Protocol, Parallelism, Distribution and more Single cost implementation Simple set of Get/Set operations specific to the implementation TD TD TD Uses full matrix communications in first release Blocks of Tuples are distributed round robin Full Communication Matrix Session Data is MultiPlexed Multiple sessions use same communications channel QG2 TD TD TD Aster Aster Aster Aster Current Connectors

39 Aster - Foreign Server Syntax Support DML syntax - external objects Teradata s Foreign Server Syntax Aster & source: Bi-directional data movement Load_from_Hcatalog, Teradata, etc Load_to_Hcatalog, Teradata, etc Use: SEL,INS,Views and CTAS Query pushdown, Query time special & override of parameters also supported Grant & Provoke USAGE & EXECUTE privileges CREATE FOREIGN SERVER name USING server( ') port('1234') DO IMPORT WITH Load_from_XYZ USING DO EXPORT WITH Load_to_XYZ USING SELECT * FROM table@foreignserver; INSERT INTO table@foreignserver SELECT id, value FROM astertable; WITH FOREIGN SERVER fsalias as (foreignserver using username('foo') password('bar') ) SELECT * FROM table1@fsalias, table2@fsalias ;

40 AsterX 7.0 Scripting Pattern Changes Existing Customers - Implementing persistence in AX Best practices in script writing Disable Failure mode until after DDL commands Truncate Tables (delete from all) Create Tables inline (keeps code in one place, enables operator to drop table and not have to change production code) Cascading Insert/Selects/CTAS Pour over tables for failure/locking latency Option of creating a cluster sized just for this workload

41 Aster 7.x Other Cool Stuff

42 AsterX 7.0 Planner Changes Improve plans for external table (ET) queries External tables are the norm in AX (exception in AD) 7.0 Planner Hive Meta-store to get table size AD 6.50 planner view of ETs AX 7.0 planner view of ETs??? 3,000,000,000 6,000 4 Region Sales Store Region Store Sales

43 AsterX 7.0 Planner Changes Planner Hive Meta-store to get base table stats Only table rowcount and size. No columns/histograms Recognize small ETs and replicate as dim tables Save on costly data repartitioning Improve join order optimization Avoid early theta joins and dataflow multipliers Better skew avoidance Avoid partitioning on low cardinality columns

44 AsterX 7.0 Multi-Tenancy User Edge Node App B Queen User Edge Node App A Queen Head Nodes Hadoop Services Name Node Hadoop is an Execution Environment Aster must conform to Hadoop s capabilities Hadoop supports Sessions and Aster supports Sessions Ergo how does Aster run inside Hadoop... Aster is a Daemon based architecture... Aster A Aster B Aster B Hive Aster A Aster A Map/Reduce Worker Nodes Multi-Tenancy in AX7.0 Co-exist with other Hadoop Applications Port Mapping is the largest single problem

45 Thank You Questions/Comments Follow Me DataOcean Rate This Session # 598 with the PARTNERS Mobile App Remember To Share Your Virtual Passes 45

46 Aster 7.10 What s NEXT (where s the cool graphic???)

47 Aster 7.10 Containers (using Docker) Objective: Architecture using Containers (what processes go where) - Hadoop Major Impact to Startup, Distribution, Process Management Reality: Extraordinarily difficult on Hadoop Required complete rewrite of all process management Theory issue problems Process Allocation and management Foundation of AsterX on other platforms - GCP, AWS, Azure... Open decision on long term Hadoop implementation

48 Aster 7.10 Planner Improvements Pushdown predicate to Hive Automatic in 7.10 (manual in 7.0) Reduces data movement Utilizes store format filters Cuts down IO & CPU Filter Increase dependence on Stats Foreign System Get/Set Scan Scan + Filter

49 Aster 7.10 Planner Improvements Planner pushdown sub-queries to Hive Intelligent push down (semantics, type, size) Minimal data movement Better stats & data distribution utilization NPath NPath T4 GB T4 ET1 ET2 ET3 GB ET1 ET2 ET3

50 Aster 7.10 Aster Spark Utilize Spark execution framework for Aster Aster query operator as Spark functions / scripts Uses Spark MLlib analytics libraries Customers write functions in Spark using Uses familiar SQL/MR language framework Support multiple Spark clusters (ex: same query) Parallel data transfer (Sockets or ) Spark Job Monitoring

51 Aster 7.10 Spark Aster Aster table/queries use Spark Data frame API Read Aster tables/queries in parallel Can cache data on disk sqlcontext.readastertable ( <table-name>, cache-on-disk>, ) sqlcontext.readasterusingquery ( <query>, <cache-on-disk>, ) Write Data frames Tables in parallel (overwrite / append mode) <dataframe>.writetoastertable( table-name, <mode>, )

52 Existing Framework (Analytic flow) Batch-mode Processing Training Data Queries (Test Data) Appropriate Action Analytics (Model Builder) Model Analytics (Predictor) Prediction Requests Prediction Response Score ASTER FRAMEWORK

53 Proposal: Split the processing Real-time Scoring Aster Platform - Prediction analytics isolated from training (modeling) RealTime Platform -Asynchronous feedback between the two frameworks. Training Data Queries (Test Data) Appropriate Action Analytics (Model Builder) Model Prediction Requests Analytics (Predictor) Prediction Response Score

54 Aster Model Language Generator Generates AML File from Model Table Training Data DRIVER FUNCTION Real-time Scoring Queries (Test Data) Appropriate Action Analytics (Model Builder) Model AML Generator Prediction Requests Analytics (Predictor) Prediction Response Score

55 Scorer Execution Flow AML File Your Real Time Framework Model Type Model Definition Model Data Request Parameters Request Definition Transport as Request Prediction RequestsAppropriate Action Java JAR file Response Configurator Score Prediction Response