INNOVATION CAMP July 18 & 19, 2018 SAP HQ

Transcription

1 SAP Digital Business Services INNOVATION CAMP July 18 & 19, 2018 SAP HQ Next Generation Data Management Digital Platform Track Mrinal Sarkar Support Architect Global CoE NA SAP Digital Business Services Abhishek Kumar HANA Architect Global CoE NA SAP Digital Business Services PUBLIC

2 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 2

3 Introduction Simplified Data Management Introduction Efficient Data Management strategy important for customer. Keep actual data in memory and historical data on disk. Not all data has the same value. Because of cost and performance reasons many customers want to move historical data to a different storage tier. In this session we describe different options to move data to different storage tiers. 3

4 Data Volume Introduction Data Growth Data growth dilemma Typically only ~20% of company data are operational Strong dependencies between database size and HW costs in SAP HANA Scalability may saturate The data growth challenge Decouple data growth from CPU/memory Analyze your data (value, growth rate) Analyze your applications (data access, SLAs) Evaluate your hardware (limitations, extensibility, scalability) Define your Data Volume Management from the beginning Data Growth Dilemma Data Growth Hardware Costs Add HW Nodes Time 4

5 HANA Introduction Multi-Temperature Data Management HANA data tiering is the assignment of data to storage classes/media based upon data type, operational usefulness, performance requirements, frequency of access, and security requirements of the data. Hot Store This tier is used to store mission-critical data for real-time processing and real-time analytics. Data is retained in-memory of the SAP HANA database. Warm Store This tier is used to store data with reduced performance SLAs, which is less frequently accessed. Data is stored on a lower cost storage tier, managed as a unified part of the SAP HANA database. Cold Store This tier is used to store voluminous data for sporadic or very limited access. Data is stored on low cost storage tiers, like disk or Hadoop, managed separately from the SAP HANA database, but still accessible at any time. 5

6 Introduction SAP HANA Data Tiering Technical Layer HANA In-Memory Disk Extension Nodes Hybrid LoB Page Attributes Extended Store Dynamic Tiering SAP IQ SAP IQ Hadoop, File System SAP Data Hub Archiving 6

7 Introduction SAP HANA Data Tiering Application Layer HANA S/4HANA BW/4HANA BW on HANA Native HANA In-Memory Extension Nodes Extension Nodes (1) Disk Data Aging Extended Store Dynamic Tiering SAP IQ DTO/ NLS Hadoop, File System Archiving DTO/ NLS SAP DWF / DLM (2) (1) Piloting Phase, see Note (2) DLM 2.0 doesn t support SAP IQ as a target anymore 7

8 Introduction SAP HANA Data Tiering Overview about the Use Cases and Storage Options Tier Data Age Storage Option 0 Actual SAP S/4HANA or Business Suite on HANA SAP BW on HANA or BW4/HANA SAP HANA Native SAP HANA In-Memory SAP HANA Dynamic Tiering 1 Actual/Historical SAP HANA Extension Nodes 2 Historical Data Aging (Next Gen ILM) Near-line Storage (NLS) Data Archiving (ADK) General available Combination not available 8

9 Introduction SAP HANA Data Tiering Overview about the Use Cases and Storage Options Tier Data Age Storage Option OLTP OLAP Accessibility Performance (OLTP/OLAP) Storage Costs Implementation Complexity Hot Warm Actual Actual/Historical SAP HANA In-Memory Good Very Good/Very Good High Low SAP HANA Dynamic Tiering Data Aging (Next Gen ILM) Good (Same Data Model) Good (Same Data Model) Slow/Good Medium Medium Good/Slow Medium Low Cold Historical Data Archiving (ADK) Hadoop Limited (Different Data Model) Good (Same Data Model) Slow/Slow Low High Slow/Slow Low High 9

10 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 11

11 HANA Dynamic Tiering Use cases for DT in HANA solutions SAP HANA native data marts / warehouses or applications Add option to manage HANA memory footprint for native scenarios Evaluate use of DLM tool (from SAP Data Warehousing Foundation) see Documentation Table and data management: responsibility of data mart architect / application developers Verify whether your required advanced functionalities are supported with extended or multistore tables (e.g. predictive, geospatial, text ) SAP HANA SAP BW on HANA makes limited use of dynamic tiering: Dynamic tiering can be used for Write-optimized DSOs and advanced DSOs in the propagation and transformation layers of SAP BW (7.40 SP 8 and above) SAP recommends to use extension nodes for warm data management in BW SAP BW/4HANA currently does not support SAP HANA dynamic tiering SAP Bank Analyzer (OLTP Workload) Financial results for previous years could be sent to DT 12

12 HANA Dynamic Tiering Architecture HANA Dynamic Tiering Architecture Dynamic Tiering (DT) option adds to SAP HANA a disk-based extended store. Extended Store (ES) with Dynamic Tiering (DT) integrates SAP IQ technology into SAP HANA. It is not an IQ database installation, but SAP HANA Dynamic Tiering component installation. Extended Store (ES) uses persisted indexes to accelerate data access, and loads pages with currently accessed data into a buffer cache in memory. Dynamic Tiering is fully integrated into SAP HANA and appears as one database management system with the extended store as an additional disk-based data store. The Extended Store is managed by a separate server process in the SAP HANA system, the extended store server (ES server). It is running besides the Index server process for the In-Memory component. (See slide after next) The Extended Store is fully integrated into database backup and recovery. Backup and recovery includes the whole database, with tables in the extended store and the in-memory stores. Incremental and differential delta Backups supported for dynamic tiering since HANA 2.0. Since HANA 2.0, two-tier HANA synchronous system replication is integrated with DT node in the landscape. 13

13 HANA Dynamic Tiering Dedicated host deployment with HANA Single Node Use Case SAP HANA Server Nodes Repository SAP HANA SAP HANA System with DT (one SID) ES Server Nodes SAP DT Lifecycle Mgr Agents Agents Master Node Standby Node Agents Agents Name Server Name Server Worker Node Standby Node Index Server Stats. Service XSA Server Index Server Extended Store Server Name Server Extended Store Server Name Server log data log data FC Storage log data log data shared NAS Storage (NFS) Minimum setup with failover scenario - For HANA single node implementation scale-out with one standby node is the only solution for local HA - ES server able to provide failover node as well (automatic failover for HANA server node & ES server node) 14

14 HANA Dynamic Tiering Same host deployment with HANA Single Node Use Case SAP HANA Server Nodes Repository SAP HANA SAP HANA System with DT (one SID) ES Server Nodes SAP DT Lifecycle Mgr Agents Agents Primary Node Standby Node Name Server Name Server Index Server Stats. Service XSA Server Extended Store Server Name Server Index Server Extended Store Server Name Server log data log log data data log data log data FC Storage shared NAS Storage (NFS) For HANA 1.0, Same host deployment can only be used for non-production systems; For HANA 2.0, Co-deployment (same host) of HANA and dynamic tiering Supported for HANA scale up systems only. Allowed for production environments, but will have some performance impact to the HANA index server. 15

15 HANA Dynamic Tiering Dedicated host deployment with HANA Scale-Out Use Case SAP HANA Server Nodes Repository Lifecycle Mgr Agents Agents SAP HANA Agents SAP HANA System with DT (one SID) Agents ES Server Nodes The hosts for SAP HANA dynamic tiering do not need to be based on hardware certified for SAP HANA(you may of course choose HANA-certified hardware) SAP DT Master Node Worker Node Worker Node Standby Node Agents Agents Name Server Name Server Name Server Worker Node Standby Node Index Server Stats. Service Index Server Index Server Index Server Extended Store Server Name Server Extended Store Server Name Server XSA Server log data log data log data log data FC Storage log data log data shared NAS Storage (NFS) ES server only supports one active node plus a failover node (no scale-out supported) ES server node must be able to access the HANA shared file system FC LUNs attached via HANA Storage Connector Interface 16

16 HANA X Customer (Without DT) with 48 TB HANA single node HANA Sizing Without DT & 20% Growth Projections as per Feb-2018 Sizing Exercise Sizing Summary HANA 24 TB (50% of Hardware capacity) would be reached by Jul TB Data Reaching Time 32 TB Data Reaching Time 17

17 HANA X Customer (With DT) with 48 TB HANA single node HANA Sizing With DT & 20% Growth Sizing Summary HANA 24 TB (50% of Hardware capacity) with DT solution will be reached by Aug-2019 With DT we get an additional year with current HANA hardware. 24 TB Data Reaching Time 18

18 Technical Architecture and Infrastructure Summary Technical Landscape For DT DT Customer : P r o d Name Server Index Server Stats. Service XSA Server Primary Node (AFP) Extended Store Server Name Server Name Server Index Server Stats. Service XSA Server HA Node (AFP) Extended Store Server Name Server Name Server Index Server Stats. Service XSA Server DR Node (AFP) Extended Store Server Name Server log data log log data data log data log log data data log data log log data data P r e - P r o d Name Server Index Server Stats. Service XSA Server Primary Node - PPS Extended Store Server Name Server Name Server Index Server Stats. Service XSA Server HA Node (PPS) Extended Store Server Name Server Planned Setup With DT: Total Hardware Size: 48 TB / 32 sockets / 576 cores HANA DB: 42 TB / 28 sockets / 504 cores DT DB: log data log log data data log data log log data data 6 TB / 4 sockets / 72 cores This is the current state of planning and may be changed by SAP at any time. 19

19 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 20

20 Data and system management with dynamic tiering Data Management State as of HANA 2 SPS 03: Application controlled data aging Multistore tables with time selection partitioning, and extended tables Tool support using DLM Cross-store query optimizer, including Calculation Views Encryption of the dynamic tiering store Not supported at this time: Scale out for larger data volumes Multistore table Backup & Recovery State as of HANA 2 SPS 03: Integrated backup (whole system) Full point-in-time recovery Full, file-based backup Delta backups supported Backint interface implemented Full backup encryption Not supported at this time: Storage snapshots HANA Index server Name server HA/DR State as of HANA 2 SPS 03: Automatic failover Two-tier a/synchronous and three-tier system replication Not supported at this time: Cluster managers Data Center 1 HANA host Primary (active) DT host Data Center 2 Secondary (active/active for HANA data) HANA host DT host Hot_P1 Hot_P2 Warm_P3 Warm_P4 Dynamic tiering XS engine DT table spaces One Data backup Data / Log Data / Log Data / Log Data / Log 21

21 SAP HANA dynamic tiering Multistore table new for HANA 2.0 Add-on option to SAP HANA Manage data of different temperatures Hot data (always in memory) classical HANA Warm data (disk based data store) Introducing two new types of database table: Extended table disk-based columnar table with all data on disk Multistore table HANA partitioned table with some partitions in memory, and some on disk Extending the SAP HANA database Deep integration Common installation, monitoring, administration Data backup and system replication for high availability Consistent transaction management Transparent query processing & optimization Encryption for data security Target scenarios Data warehousing, analytical applications SAP HANA Database In-memory-store Extended store Column table T_C Multistore table T_M Partition 1 Partition 2 Partition 3 Extended table T_E 22

22 Heterogeneous multistore table partition by range on two columns New for HANA 2.0 Flexible partitioning: first-level partitions may have different second-level partition ranges Supported for HANA column tables as well Partitioning columns do NOT need to be part of primary key The first level partition may reside in either default or extended storage (second-level resides in the same store as first-level) Only range-subrange partitioning is supported Example (simplified syntax): CREATE COLUMN TABLE (P INT, S INT, ) PARTITION BY RANGE (P) PARTITION P_1 USING DEFAULT STORAGE SUBPARTITION BY RANGE (S) (S_A, S_B, S_C) PARTITION P_2 USING DEFAULT STORAGE SUBPARTITION BY RANGE (S) (S_D, S_E) PARTITION P_3 USING EXTENDED STORAGE SUBPARTITION BY RANGE (S) (S_F) PARTITION P_4 USING EXTENDED STORAGE SUBPARTITION BY RANGE (S) (S_G, S_H) PARTITION P_5 USING EXTENDED STORAGE You can specify that some partitions are read-only by specifying the INSERT ON/OFF clause Memory P_1 P_2 The PRIMARY KEY UPDATE clause specifies whether you can update primary key columns S_A S_B S_C S_D S_E Useful for tables that don t have evenly distributed data, and have empty partitions. Disk P_3 P_4 P_5 HANA supports only partitions in a table S_F S_G S_H 23

23 Dynamic tiering to HANA shipping optimization Compressed ITAB (New for HANA 2.0) ID Name Brand Mnfctr Price P1 Prod1 XYZ ABC Dynamic tiering sends dictionary compressed result as a BLOB, instead of sending uncompressed results row by row. P2 Prod2 XYZ ABC P3 Prod3 AAA ABC P4 Prod4 BBB ABC P5 Prod5 CCC JKL Big advantage when results are returned from tables that are highly compressed in dynamic tiering. HANA keeps the data result compressed in memory, and calls functions to acquire the values from the compressed ITAB. Allows much larger result transfer which was not possible before due to memory usage. Advantages 2-3X faster than SP02 HANA P6 Prod6 AAA JKL 5.00 P7 Prod7 BBB JKL 8.99 SP02: 7 rows, 7*10 + 5*7 + 3*7 +3 *7 + 7*8 = 203 bytes ID Name Brand Mnfctr Price P1 P2 P3 P4 P5 P6 P7 Prod1 Prod2 Prod3 Prod4 Prod5 Prod6 Prod7 XYZ AAA BBB ABC JKL DT 2-5X memory saving SP03: 1 rows, 7*2 + 5*7 + 3* * *8 = 137 bytes 24

24 Intermediate result caching Performance optimization for cross-store join operations (New for HANA 2.0) HANA DYNAMIC TIERING Cross-store query optimization: Now turned on by default DT will pull data from in-memory HANA tables during cross-store JOIN operation, and cached Cache can be re-used by subsequent query requests if subset of columns and predicates are subsumed by cached query. Configuration parameters allow users to control cache size and usage by individual result sets Use of cache will be visible in esserver trace files, and also IQ query plan PRODUCTS table in memory ID Name Brand Mnfctr Price P1 Prod1 XYZ ABC P2 Prod2 XYZ ABC P3 Prod3 AAA ABC P4 Prod4 BBB ABC P5 Prod5 CCC JKL P6 Prod6 AAA JKL 5.00 P7 Prod7 BBB JKL 8.99 Query 1 SELECT P.Name, P.Brand, P.Mnfctr, P.Price, O.Customer, O.Date Query 1 pulls data from HANA FROM PRODUCTS P, ORDERS O WHERE P.ID = O.Product and P.Mnfctr = ABC Query 2 SELECT Query 2 uses result cache P.Name, P.Brand, O.Customer, O.Date FROM PRODUCTS P, ORDERS O WHERE P.ID = O.Product AND P.Brand = XYX AND P.Mnfctr = ABC ORDERS table on disk ID Customer Date Product Quantity O1 C1 1/1/2017 P1 3 O2 C1 1/1/2017 P2 2 O3 C2 1/1/2017 P3 5 Result cache ID Name Brand Mnfctr Price P1 Prod1 XYZ ABC P2 Prod2 XYZ ABC P3 Prod3 AAA ABC P4 Prod4 BBB ABC

25 HANA dynamic tiering security features Encryption capabilities (New for HANA 2.0) The dynamic tiering store may be encrypted, similar to the SAP HANA store Dynamic tiering store uses strong AES encryption with 256-bit page encryption key Dynamic tiering uses the same data volume encryption root key as SAP HANA SAP HANA and dynamic tiering stores have their own database page encryption keys Backup encryption: Dynamic tiering full backup may be encrypted Dynamic tiering incremental and log backups have the same encryption state as the HANA database HANA Server HANA Server Secure Store in File System (SSFS) SAP HANA System HANA Server Storage subsystem Encrypted hot data DT server DT server Encrypted warm data 26

26 HANA dynamic tiering supports full array of backup capabilities Full and delta backups with point in time recovery (New for HANA 2.0) HANA dynamic tiering supports: Full data backup Delta backups: Differential backup (delta since last full backup Incremental backup (delta since last delta backup) BACKINT with a single database setup (interface to third party backup tools) Redo log backups Recovery of a full data backup with no log or delta backups Recovery to point in time using full, delta and log backups ( latest time uses log area, too) Backup and recovery using either HANA Studio or HANA Cockpit HANA dynamic tiering does not yet support: Storage snapshots 27

27 HANA system replication (HSR) with dynamic tiering Two-tier asynchronous and three-tier replication (New for HANA 2.0) Data Center 1 Data Center 2 Two-tier a/synchronous and three-tier replication supported Delta store (RLV) may be enabled on primary only for 2- tier synchronous HANA Server data / log Primary DT Server data_es / log_es A/Synchronous log shipping INITIAL full data shipping HANA Server data / log Secondary DT Server data_es / log_es Supports logreplay and logreplay_readaccess operation modes only Active/active is supported on secondary site, but dynamic tiering data will be excluded from query results (operation: logreplay_readaccess) Data Center 1 HANA Server data / log Primary DT Server data_es / log_es Synchronous log shipping INITIAL full data shipping Data Center 2 - HA HANA Server data / log Secondary DT Server data_es / log_es Asynchronous log shipping INITIAL full data shipping Data Center 3 - DR HANA Server data / log Tertiary DT Server data_es / log_es Dynamic tiering may be added to an already replicating HANA system without disrupting replication Supports arbitrary number of tenants configured with DT New registration of secondary site requires full data shipping for DT. HANA can use backup or snapshots for optimized initialization of memory store. DT now supports delta synchronization of failback site Cluster managers not yet supported Takeover of a non-primary site causes DT server to be restarted no hot standby Optional asynchronous restart available (HANA does not need to wait for DT restart during takeover) Does not support near-zero downtime (NZDT) upgrade Proxy monitoring views can see HANA memory store only, not DT 28

28 Dynamic tiering support on cloud and virtualized environments The machines of a HANA system must be running on either all bare metal machines, or all virtual machines. No mixed mode allowed at this time. Virtualization support of dynamic tiering: Internal quality assurance of dynamic tiering is performed on VMWare and OpenStack virtual machines Dynamic tiering is running in HEC on virtual machines running XEN hypervisor Public cloud support (dynamic tiering must run on separate machine from SAP HANA): Dynamic tiering is supported on Amazon Web Services Dynamic tiering will announce support for MS Azure in Q SAP Note and blogs:

29 SAP HANA 2.0 SP03+ dynamic tiering Product road map overview - key themes and capabilities Recent innovations 2019 Planned innovations 2020 Product direction Deployment and administration New system views for better diagnostics Encryption of backups High availability and disaster recovery BACKINT performance improvements: no materialization to file system during recovery; parallel operation with multiple pipes log_mode = overwrite support Functional enhancements and performance Heterogeneous multistore tables for tables with uneven distribution of data Query performance improvements: zone maps, dynamic partition pruning, intermediate results caching, and data shipping optimizations Automatic data statistics maintenance Deployment and administration Support administrator connection to dynamic tiering server Additional alerts for critical events High availability and disaster recovery Estimate backup size in advance Cluster manager validation: engage with vendors to ensure proper operation of their solution with dynamic tiering Functional enhancements and performance 7-digit TIMESTAMP support Additional statistics maintained for improved query performance Partial indexing - user can choose to create indexes only on the HANA side of a multistore table Deployment and administration Multiple dynamic tiering servers allowed on a single machine High availability and disaster recovery Complete system replication support for delta-enabled extended and multistore tables Active/active support: read-only queries allowed on dynamic tiering data Functional enhancements and performance Dynamic tiering server scale out Asynchronous table replicas in dynamic tiering SAP HANA 2.0 SPS 03 This is the current state of planning and may be changed by SAP at any time. 30

30 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 31

31 New Warm Data Management for SAP BW Extension node concept in HANA scale-out landscape Master 1 TB data i i Slave 1 1 TB data Slave 2 1 TB data SAP BW HANA Standard Group: normal BW sizing, CPU/RAM ratio HOT data location Slave 3 1 TB data In order to implement the extension node concept in SAP BW, refer to SAP Note : Find steps for configuring an extension node setup in SAP Note : Extension node 2 TB data Standby node Extension node for warm data: relaxed sizing requirements (higher data/ram ratio) Enhanced Data Lifecycle Management for Warm Data Easy to set up and significantly reduced administration effort Support of all SAP HANA features for operations, updates and data management Typical Landscape Characteristics Usage of standard SAP HANA nodes Simplified sizing formula Optimized RAM/CPU ratio for warm data (Runs as asymmetric Scale-Out cluster) * Differentiation between hot and warm via BW application (definition of Object Groups) Supported BW objects: PSA, write optimized DataStore Objects, advanced DataStore Objects * Planned This is the current state of planning and may be changed by SAP at any time. 32

32 HANA Extension Group Deployment Options Master 1 TB data HANA with extension node simple example 2 TB hosts Slave 1 1 TB data Slave 2 1 TB data Slave 3 1 TB data Standby node Extension node max 2 TB data HANA with extension node advanced example 2 TB hosts Master 1 TB data Master 1 TB data Slave 1 1 TB data Slave 2 1 TB data Slave 3 1 TB data Standby node Extension node max 4 TB data HANA with extension node special example 2 TB hosts Slave 1 1 TB data Slave 2 1 TB data Slave 3 1 TB data Standby Hot Extension node n TB RAM max 2 x n TB data Standby Extension n TB RAM Option 1 ( simple ): Assign an existing node as extension node Capacity of extension node: data <= 100% RAM minimal re-configuration required Ex.: 4 TB 5 TB capacity increase (ignoring master) Option 2 ( advanced ): Assign an existing node as extension node. Capacity of extension node: data <= 200% RAM Re-configuration on storage&i/o level may be necessary (HW partner dependent) Ex.: 4 TB 7 TB capacity increase (ignoring master) Option 3 ( special ): * Add a special Extension Node with specific I/O, Storage, RAM setup HW partner offerings differ Own Standby for Extension node required model warm data & re-distribute data Ex.: 3 TB 4 + 2xn TB capacity increase Note: The examples show only one extension node, but the same is possible for more than one node. But keep in mind that only data can be classified and sized as warm that fulfils the BW restrictions. Data distribution of more then ~50% in warm are not realistic! 33 Open for customers * Planned offering

33 Differences between HANA dynamic tiering, and HANA extension nodes In-memory-store Column table SAP HANA dynamic tiering SAP HANA Database Partition Partition Multistore table Extended store Partition Extended table SAP HANA extension nodes HANA with extension node example 2 TB hosts Master 1 TB data Slave 1 1 TB data Slave 2 1 TB data Extension node max. 4 TB Overview Integration of columnar disk store technology with the SAP HANA database for warm data management Recent innovations Faster backups Performance optimizations for cross store queries New system views for improved diagnostics When to use Native HANA SQL data marts Handles data volumes up to 100TB compressed on commodity hardware Some functional gaps compared to a pure HANA system Available for native HANA applications Available for select SAP applications (SAP CAR, SAP ITOA, SAP ME) Overview Allocation of a HANA node with relaxed RAM/CPU requirements for warm data management Recent innovations Part of SAP BW s Data Tiering Optimization (DTO) framework: Unified concept covering hot, warm, and cold data Automatic displacement to warm store or cold store When to use Recommended for warm data for SAP BW on HANA and BW/4HANA No functional gaps compared to a pure HANA system Available for native HANA applications (consulting assistance recommended to ensure proper use) 34

34 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 35

35 Modern Landscapes Big Data is transforming customer landscapes From centralized, relational, on premise DWH approaches to modern distributed (cloud) Data Platforms Characteristics App.Server & DB OS & Hardware ETL Driven Structured Data Key Drivers: ERP BI / Analytics Data Warehouse FILES Structured Data / ETL DB Self Services / Analytics / ML BW/4 HANA Data Center S4 HANA SAP HANA Platform Pub. Clouds Apps SAP Cloud Streams / IoT / Web / Struct. Data Challenges for traditional architectures due to multi-structures, large data volumes, landscape scale outs Growing Cloud / Data Lake / IoT Adoption Characteristics Serverless Computing Containerized Software Distributed Data Data Driven Any Format 36

36 SAP Data Hub Current Architecture View SAP Data Hub Application SAP HANA, XS Advanced Model Platform Services UAA SDI Git Distributed Runtime Kubernetes Cluster Data Storages Cloud / On-Premise SAP Vora Containerized Relational Graph Cloud Stores AWS S3, GCP GCS, Azure ADL & WASB Metadata Catalog DB Engines Data Discovery & Profiling Time-Series Document Hadoop HDFS (optional) Scheduling & Monitoring Data Pipelines SAP Data Hub Pipelines Serverless infrastructure Scripting (JS, Python) Templates Flow-based applications SAP Data Hub Adapter Built-in Connectors Custom Operators Access Policies VORA Spark Extensions Remote Orchestration SAP Data Hub Flowagent Connectivity Connections Connected Systems SAP Integration & Open Connectivity SAP LT Replication Server Configurations Replication Jobs SAP BW Process Chains Data Warehousing Processes SAP HANA SDI Flowgraphs Data Integration into SAP HANA SAP Data Services Data Services Job Heterogeneous Landscapes 3 rd party, Open Source Direct Connectivity Messaging, APIs 37

37 38

38 Agenda Introduction HANA Dynamic Tiering o Key Features of DT With HANA 2.0 HANA Extension Nodes Data Hub Appendix 39

39 Data Movement

40 SAP Data Warehousing Foundation Data Lifecycle Manager (DLM) Define a data temperature (tiering) management strategy with DLM Leverage SAP HANA tables (Hot-Store), HANA Extension Node (Warm-Store), SAP HANA Dynamic Tiering (Warm-Store), SAP Vora*, Hadoop or SAP Sybase IQ (Cold-Store) in SAP HANA native use cases with a tool based approach to model aging rules on tables to displace aged data to optimize the memory footprint of data in SAP HANA. * planned Q

41 SAP Data Warehousing Foundation Data Lifecycle Manager (DLM) Data Movement Orchestrate and optimize the HANA memory footprint of data in SAP HANA tables Data Modification on primary Application table (e.g Hot- Store) - only on records in specific current / open periods Aged Data within closed periods to be archived / displaced to another Storage Destination Define Data Movement rules (in and out) to displace data between HANA-, Extended-, Hadoop-, SAP Vora or SAP Sybase IQ-tables Data Movement rules generated into HANA Stored Procedures to perform mass data movement Execution of HANA Stored Procedures using HANA tasks (Manual and Scheduled execution) Selective data deletion for proper housekeeping with DLM 42

42 SAP Data Warehousing Foundation Data Lifecycle Manager (DLM) HANA Extension Node Orchestrate and optimize the HANA memory footprint of data in SAP HANA partitioned tables Partitioned Column-Store table with Partitions located in Hot- Store and HANA Extension Node Low to No effort to integrate with existing Applications existing tables remain unchanged / stable existing In-Memory / Column-Store table to altered to partitioned table DLM specified aging rules to move complete Table-partitions between Hot-Store and HANA Extension Node No DLM generated SAP HANA View (Pruning / UNION) required, due to single partitioned table Data access managed by HANA incl. Partition pruning No impact to data update / delta handling, as records are moved to unique table-partition, based on partitioning criteria 43

43 SAP Data Warehousing Foundation Data Lifecycle Manager (DLM) HANA Multi-Store table Orchestrate and optimize the HANA memory footprint of data in SAP HANA Multi-Store tables Multi-Store table with Partitions located in Hot-Store and HANA Extension Node Low to No effort to integrate with existing Applications existing tables remain unchanged / stable existing In-Memory / Column-Store table to altered to partitioned table DLM specified aging rules to move complete Table-partitions between Hot-Store and Dynamic Tiering Node No DLM generated SAP HANA View (Pruning / UNION) required, due to single partitioned table Data access managed by HANA incl. Partition pruning No impact to data update / delta handling, as records are moved to unique table-partition, based on partitioning criteria 44

44 Additional Info On Extension Nodes

45 Table data Extension Nodes Deployment Options Extension Nodes Deployment Option 1: Simple Simple deployment option (with up to 100% table footprint): n TB 0.5 * n TB RAM size Working memory log data Legend master worker standby Extension Table data log data log data shared log data log data log data FC Storage NAS Storage (NFS) Minimal re-configuration required, as extension node has same hardware as other slave nodes (size of CPU, RAM, Storage). Extension node has relaxed static memory and dynamic working memory ratio requirement (compare with 1:1 ratio of master/slave nodes). Maximum size of table data distributed on Extension node is ntb (assume extension node has ntb RAM) (e.g. 2 TB extension node 2 TB table footprint) Enablement steps: model warm data, and redistribute data. Generally released since the Datacenter Service Point (DSP) of HANA SPS12. 46

46 Table data Extension Nodes Deployment Options Extension Nodes Deployment Option 2: Advanced n TB Working memory 0.5 * n TB Table data RAM size Legend master worker standby Extension log log log log log data data data data data shared log data FC Storage NAS Storage (NFS) Advanced deployment option (with up to 200% table footprint): Extension node has same RAM size as other slave nodes; But need to re-configure storage and I/O level (hardware vendor dependent). Extension node has relaxed static memory and dynamic working memory ratio requirement (compare with 1:1 ratio of master/slave nodes). Maximum size of table data distributed on Extension node is 2*nTB (assume extension node has ntb RAM, and not all the tables are loaded into memory). (e.g. 2 TB extension node 4 TB table footprint) Enablement steps: model warm data, and redistribute data. Generally released since the Datacenter Service Point (DSP) of HANA SP12. 47

47 Extension Nodes Deployment Options Hardware Configurations for Option 1 + 2: Since SAP HANA Extension Node setups with Option 1 and Option 2 use standard HANA HW configurations ALL hardware partners have matching offerings (the configuration of an SAP HANA Extension Node is then done following the guidance in SAP Note /SAP Note ). Some partners may have offerings that go beyond this with special configurations please check this with your hardware contacts. 48

48 Additional Info On Data Hub

49 SAP Data Hub Architecture View SAP Data Hub delivered as XSA application, leveraging SAP HANA as runtime environment and metadata store Processing data is stored in e.g. Hadoop, Data Lakes, object storage The SAP Data Hub Distributed Runtime based on SAP Vora is a fully distributed, containerized big data infrastructure. It is comprised of shared services and tools (for example, catalog and tools), and distributed engines (for example, relational engine, and so on). It is installed on an existing Kubernetes infrastructure. Definition of processing logic in Data Hub, execution via Agent and Vora in cluster environments SAP Data Hub Cockpit following Fiori paradigms, implemented in SAPUI5 SAP Data Hub Modeling Perspective embedded into SAP Web IDE SAP Data Hub Vora Spark Extensions are installed on each Spark worker node of the connected Hadoop cluster, and provide access to SAP Vora data sources from a Spark environment. For details about the support Hadoop distributions, Spark versions and operating systems, see the Product Availability Matrix (PAM). SAP Data Hub SDI Adapter runs as a plugin to the SAP HANA Data Provisioning Agent on one node (edge node) of a connected Hadoop setup. It serves as the central communication endpoint for all operations executed on the SAP Data Hub Distributed Runtime for the SAP Data Hub application tier. For details about the supported Hadoop distributions and operating systems, see the Product Availability Matrix (PAM). 50

50 SAP Data Hub Data Orchestration & Data Pipelines SAP DATA HUB application Enterprise Applications SAP BW SAP HANA Metadata Catalog Data Discovery & Profiling Scheduling & Monitoring Data Pipelines Access Policies SAP Data Services Data Pipeline Data Pipeline Execute Pipeline SAP BW Big Data Cluster & Object Stores Load Data into Data Lake Change File Formats SAP Data Hub Process Chain Execute Pipeline SAP DATA HUB runtime Workflow Orchestration Data Hub Pipelines Data Integration, Streaming, Messaging Listen KAFKA Write HDFS Cleanse Data Execute Python Persist Result SAP Data Services Kafka Data Storage (Hadoop, S3, GCP, Azure, BDS) 51

51 Landscape X Customer Vora 2.1 (Deployment Option) Data Sources DS 4.1 SP6 AIX 6.1 Hadoop 2.7 is required Power BI Win2012 R2 ArcGis Win2012 R2 Hadoop DN DN DN HD HD HD HDFS SAS Guide Win2012 R2 Spark Extns VM Server Views BOE Dashboard AIX 7.1 HANA Live DB Tenant M Node M Node Lumira 1.31 Win2012 R2 BW BCP (PCP) DB Tenant K8 s Vora 2.1 M M 250M C.1 W Node W Node W Node W Node VM Server TDF DB Tenant SAP HANA 2.0 SAN SAN ECC DB Tenant HANA 2.0 is recommended. SLT Connection to SAP HANA using SAP HANA smart data access (SDA) using the SAP Vora remote source adapter voraodbc Persisted data is not required in Vora 2.1, Data is read from HDFS On the left it is shown a possible deployment option with Vora 2.1 HANA 2.0 is recommended, earlier HANA version may have some performance constrains HANA access Vora tables using federation, virtual tables Vora can keep in memory tables that will be access by the upper layers, and it is possible to created analytical scenarios using the views from Hana Live and Vora tables 52

52 SAP Data Hub Distribution Runtime Landscape X Custom Data Hub (Deployment Option) Hadoop 2.7 is required Power BI Win2012 R2 ArcGis Win2012 R2 Advanced Data Modeling with SAP HANA SAS Guide Win2012 R2 VM Server Data Hub Applications XSA Data Provisioning Agent DB Tenant BOE Dashboard AIX 7.1 HL Views ECC Replicated Tables DLM Lumira 1.31 Win2012 R2 SAP HANA 2.0 BW BCP (PCP) DB Tenant In general connection from Lumira to Data Hub is possible, however it has not been 100% tested. Refer to PAM TDF DB Tenant ECC DB Tenant HANA 2.0 is recommended. SLT Data Hub 1.0 is currently available with Vora 2.1. Newer Vora versions are planned for this year and the release will be announced via PAM. With Data Hub 2.0 we can expect the same basic architecture including Vora as depicted in this deployment option. Data Sources Kafka Hadoop (HDP 2.7) DN DN DN HD HD HD HDFS Data Hub Adapter Spark Extns Spark Vora 2.1+ M Node M Node M M 250M C.1 W Node W Node W Node W Node K8 s VM Servers Data Hub Pipelines Thriftserver Storage (Local & NFS, HDFS) Data pipelines can interact with different technologies, e.g. SAP HANA, SAP API Business Hub, Kafka, any web service. Direct connection from applications via SAP Thriftserver is not recommended. BI Tools should connect to Vora through HANA 53

53 Thank you. Contact information: Mrinal Sarkar IT Planning CoE E: Contact information: Abhishek Kumar HANA CoE E: Partner logo