SAP HANA Data Warehousing Foundation Data Distribution Optimizer / Data Life Cycle Manager DWF SP03 February, 2016 This is the current state of planning and may be changed by SAP at any time.
Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 2
SAP HANA Data Warehousing Foundation Tools in detail Data Distribution Optimizer Data Lifecycle Manager Data Warehouse Monitor ** Data Warehouse Scheduler ** SAP HANA Data Warehousing Foundation DWF 1.0 SP00: DDO >= HANA Rev. 83 DWF 1.0 SP01: DDO+DLM >= HANA Rev. 96 DWF 1.0 SP02: DDO+DLM >= HANA Rev. 101 SAP HANA Data Warehousing Foundation 1.0 delivers specialized SAP HANA XS based applications Data Distribution Optimizer to plan, adjust and analyze landscape reorganizations for SAP HANA scale out systems Data Lifecycle Manager to deliver a possibility to archive / displace data from a SAP HANA persistency to Dynamic Tiering, SAP IQ or Hadoop Data Warehouse Monitor to provide a comprehensive overview about current and past activities in the data warehouse Data Warehouse Scheduler to maintain dependencies between single processes with the focus to provision data warehouse models DWF 1.0 SP03; DDO+DLM >= HANA Rev. 110 ** Outlook 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 3
Data Distribution Optimizer (DDO) This is the current state of planning and may be changed by SAP at any time.
Data Distribution Optimizer Functionalities SAP HANA Scale Out Landscape Overview with respect to system configuration and data distribution Specify different configurations for reorganization on different levels of granularity (e.g. DB-Schema, Table, Table Group, Table Subtype, Location Group) Create, adjust and simulate different reorganization plans to achieve an optimal data distribution in an interactive fashion. Repeat the steps until a sufficient reorganization plan has been created. Graphical visualization of reorganization plans by comparing the simulated with the actual system landscape Export and import reorganization plans across system boundaries (-> DDO remote mode deployment) Execute & Schedule SAP HANA reorganization runs Analyze reorganization plan steps and logs of planned and executed SAP HANA landscape reorganization runs Join Path Analysis to identify associated database tables in the system and create a proposal for table grouping (SP03) 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 5
DDO SPS03 - Join Path Analysis (JPA) Identify associated database tables and make a proposal which are frequently used together are going to be placed together (via Table Grouping) Table Group 1 Table Group 2 Use the proposal to assign the grouping information to the database tables to significantly improve the placement of tables, so that table joins are executed with an adequate level of speed. Table 1 Table 8 Table 3 Based on the data available in the statement cache (HOST_SQL_PLAN_CACHE view of the _SYS_STATISTICS schema) JPA uses currently the execution count of a statement for prioritizing joins Only executed SQL statements that access tables or views will be analyzed. SQL statements that haven't been executed yet or view definitions are not considered for join path analysis. This means that depending on the time of execution the results of the join path analysis may differ. Table 2 Table 7 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 6
DDO SPS03 - Join Path Analysis (JPA) - (Tabular Display) Leverage the Grouping Granularity Slider: When increasing the granularity, the groups get smaller (less tables within a group) and the number of groups increases. When decreasing the granularity, the groups get larger (more tables within a group) and the number of groups decreases. The goal is to find a trade-off that has enough groups so that the system can be balanced well-enough, while at the same time the number of cross-nodejoins is minimized. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 7
DDO SPS03 - Join Path Analysis (JPA) - (Chart Visualization) Adjustment to the Grouping Granularity Slider would apply for both Tabs: The related Group(s) are highlighted when selecting a Group (here: Group 131) representing the JPA derived dependencies. Further drill-down capabilities are supported within the Bubble-Chart. The Tables + Table Size listed in the Tables in Group xxx box are derived based on the selected Group in the Bubble-Chart 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 8
Product Roadmap Overview Data Distribution Optimizer SAP DWH Foundation 1.0 SP03 Data Distribution Optimizer (DDO) Deeper insights and simulations for data distributions Expected duration Modification of proposals Improved Plan Generation runtime Workload analysis (e.g. plan cache analysis) Analysis of SQL plan cache / participating tables Improvements to the simulation of landscape reorganization runs Upcoming planned release Data Distribution Optimizer (DDO) Parametrization of JPA (e.g. SCHEMA) Integration of JPA with Plan Generation Optimal Group Granularity Detection Maintenance of Table Group assignments for native HANA Datamarts Future innovations Further enhancements of DDO Support and optimize table partitioning in native SAP HANA cases Flexible management of hosts Deeper insights and simulations for data distributions Additional KPIs Today Q4 / 2015 Planned Innovations Q2 / 2016 Future Direction 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 10
Data Lifecycle Manager (DLM) This is the current state of planning and may be changed by SAP at any time.
DLM SPS03 - Graphical Exploration Module (Drill Down, Top 10) If the utilization forecast on the Home screen indicates that the utilization of one or more nodes could become critical over time, you can use the exploration function to establish which tables are appropriate candidates for data relocation. Context With a two-step approach, you can first perform graph-based exploration on different levels. With a sunburst chart, you can drill down to information on different levels, host, schema, or table level for example. You can thus identify the ten largest tables of all the tables in the system, or the ten largest tables in a given schema on a given host for example. In a second step, you can use form-based exploration to see how data of a selected SAP HANA table is spread according to a selected column filter. With this insight, you can derive meaningful business rules for relocating the data. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 12
DLM SPS03 - Home Screen: KPIs Additional KPI: Forecast HANA: Disk Forecast HANA: Disk in 100 days vs. current utilization in addition to Forecast HANA: Memory in 100 days vs. current utilization 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 13
DLM SPS03 - Manage Storage Destinations A storage destination specifies an instance of the storage stack which data can be relocated to with Data Lifecycle Manager. Supported storage destination types are SAP HANA Dynamic Tiering, Spark SQL (DESTINATION) or SAP IQ for example. For deletion of data, Data Lifecycle Manager offers storage destination type Deletion Bin. A storage destination is assigned to a lifecycle profile. It is used there to define to which destination persistence the data will be relocated to. The Manage Storage Destinations screen of the Data Lifecycle Manager is split up into two main areas: On the left side of the Storage Destinations area, you see a list of all available storage destinations. Here, you can search for storage destinations by storage destination name or the user who last changed the destination. You can add new destinations or delete destinations that are not needed anymore. Note: The instances of the storage stack have to be installed and configured properly before you can use them for defining storage destinations. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 14
DLM SPS03 - Supported Storage Destination Types Storage Destination Type SAP HANA Dynamic Tiering Local SAP IQ over SDA Spark SQL (DESTINATION) Deletion Bin Destination Restrictions The following factors prevent lifecycle profile from being activated: Source tables with keys of type VARCHAR/CHAR/VARBINARY/BINARY and length greater than 255 Source tables with keys of type NVARCHAR/NCHAR and length greater than 85 Source tables of type row Source tables with columns with data type BLOB Source tables without a primary key The following factors prevent the lifecycle profile from being activated: Source tables with columns with data type TEXT, SHORTTEXT, BINTEXT, BLOB, CLOB, TIME, VARBINARY, NCLOB, or ALPHANUM The following factors prevent the lifecycle profile from being activated: Source tables without a primary key 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 15
DLM SPS03 - Hadoop Storage Destination Depending on the type of stores you would like to use for relocating data: The SAP HANA Spark Controller has been installed and configured, and added as Remote Source to SAP HANA. The SAP HANA system has been configured for Spark connectivity. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 16
DLM SPS03 - Hadoop Storage Destination (cont.) 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 17
DLM SPS03 - Manage Modeled Persistence Objects Modeled persistence objects help you treat multiple source persistence objects, like tables, jointly in matters of data relocation. You can define a source persistence object from multiple SAP HANA tables. Such a modeled persistence object can be used in a lifecycle profile. The relocation rule in this case can be defined for common columns, which have been selected during definition of the modeled persistence object. Note: Modeled Persistence Objects are supported for storage destination types SAP IQ over SDA and SAP HANA Dynamic Tiering. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 18
DLM SPS03 - Destination Persistence Modeled Persistence Objects Manage Lifecycle Profile Destination Persistency: Visibility of the Data Flow(s) of the defined Modeled Persistence Objects SAP HANA Table Group (here: YAM_ORDER_HDR + YAM_ORDER_ITM) 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 19
DLM SPS03 - Rule Editor: Manage Modeled Persistence Objects When you create and execute a lifecycle profile for a modeled persistence object, the following applies for all tables of the modeled object: A common relocation rule will be defined and executed. A common clash strategy and packet size applies. A single relocation procedure will be generated. For all tables corresponding target tables will be generated. An example for a modeled persistence object is a table group. A table group is a set of SAP HANA tables with common columns. Table groups are supported for storage destination types SAP IQ over SDA and SAP HANA Dynamic Tiering Local. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 20
DLM SPS03 - Import / Export of DLM Entities Data Lifecycle Manager entities like storage destinations, modeled persistence objects, or lifecycle profiles can be described in JSON format. For transport or backup purposes, you can export and import the entities in JSON format. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 21
DLM SPS03 - Lifecycle profile: Nominal key For a source persistence type SAP HANA table, you specify a schema and table name for example. Additionally, for tables without a primary key, you can specify a nominal key, which is treated like a primary key by Data Lifecycle Manager. Using the nominal key, you can activate a life cycle profile for a table without primary key. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 22
Product Roadmap Overview Data Lifecycle Manager SAP DWH Foundation 1.0 SP03 Support for Table Groups Generate DLM profiles for a predefined set of tables (table group) Graphical Exploration module for HANA Table SPARK (SQL): HADOOP as storage destination Upcoming planned release Table Hierarchy support (Foreign Key Relationship) Rule Editor enhancements (Semantic checks / Customer exit) Enhanced support for Table Hierarchies and Table Groups for certain storage destinations Feature completeness for HADOOP as storage destination (packet size, clash strategy) Adapt Relocation Agent module for Table Groups & Table Hierarchies for supported persistent storage destinations Future innovations Integration with SAP HANA Vora Pre-delivered Application DLM templates Support for multi-tiering Integrated lifecycle management Exploration module based on SQL Query Access Cache Parallelization of relocation packet processing Support for HANA CDS Entities as source persistence object Automated substitution of data foundation in Calculation Views (table replaced by generated view) Today Q4 / 2015 Planned Innovations Q2 / 2016 Future Direction 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 23
Appendix This is the current state of planning and may be changed by SAP at any time.
Additional Information on DWF 1. Data Warehousing Foundation (DWF) SAP Help Page : http://help.sap.com/hana_options_dwf 2. SAP HANA DWF in SCN: http://scn.sap.com/docs/doc-62482 3. Product Availability Matrix (PAM): https://support.sap.com/release-upgrade-maintenance/pam.htm 4. SWDC: https://support.sap.com/software/installations 1. -> A Z Index -> SAP HANA Data Warehousing Foundation 5. Support / Incident Components 1. Component HAN-APP-DWS - for all SAP HANA Data Warehousing Foundation Option issues. 2. Component HAN-APP-DWS-DDO for DDO related issues 3. Component HAN-APP-DWS-DLM for DLM related issues 6. SAP HANA Academy Channel - Data Warehousing Foundation (DWF) https://www.youtube.com/playlist?list=plkzo92owknvxzclrgmfzs-bdlifyrq39r 7. DDO System Meta-Data Extraction: SAP Note 2138117 8. SAP DWF 1.0 Release SAP Note 2092669 9. SAP DWF 1.0 SP03 DDO Patch 1: 2259749 10.SAP DWF 1.0 SP03 DLM Patch 1: 2259304 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 25
Thank you 2015 SAP SE or an SAP affiliate company. All rights reserved.
2015 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE s or its affiliated companies strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forwardlooking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions. 2015 SAP SE or an SAP affiliate company. All rights reserved. Public 27