IBM for z/os Leverage data virtualization synergy with API economy to evolve the information architecture on IBM Z IBM z Analytics
Agenda Big Data vs. Dark Data Traditional Data Integration Mainframe Data in the API Economy Q&A
Data resides in silos and unfamiliar formats Speed of business forcing need for real-time data Need for more data agility to support cloud, mobile There is simply too much data for older architectures
IBM z Analytics 4
Dark data has to be made accessible and transformed to a useable format before it can be turned into business insight
Some of the concepts explained Virtual tables to access data sources Hardware dependant optimized code base Architectural rules Parallelism / Map Reduce No Staging Health-level monitoring Virtual Parallel Data Performance
Traditional data integration inadequate Becoming no longer viable to move data (using extract, transform, load ETL) Risk to data security Data inconsistency Rigid, limits business agility High cost and latency 8
A New Approach - Data Virtualization Data virtualization enables data structures that were designed independently to be leveraged together, in real-time, without the necessity of physical data movement.
How Customers use Data Virtualization Real-time Analytics Modernization Optimization Need immediate insight into your customer or business Difficult access to data (SMF, non-relational, MQ) Faster, easier delivery of digital systems of engagement Real-time Z Data is a needed via multiple APIs Improve efficiency/performance for any application needing Z data Too costly and time-consuming for applications to get Z Data 10
Data Virtualization Technology Accelerates Real-time Analytics Universal Data Access Data Virtualization for all enterprise data on and off mainframe BI Solution with Access, Virtualization and Visualization and Data Preparation QMF for z/os Data Virtualization Manager for z/os DB2 Analytics Accelerator Loader Query Acceleration High-speed processing for complex Db2 queries Common metadata maps to share and reuse Predictive Analytics Discovering patterns/meaning in data Machine Learning for z/os Open Data Analytics for z/os 11
QMF - Data Virtualization Expands Insight into VSAM, IMS, Adabas Easy mapping wizard. Setup once, then data can be used by any QMF user Federation is virtualized in-memory in z System Exploits specific z Systems hardware and ziip specialty engine Relational DBs Non Traditional DB2 VSAM QMF DB2 LUW MS SQL Oracle Teradata Sybase PostgreSQL Excel Flat File TPF Hadoop Spark Website data IMS Adabas In-Memory Data Virtualization 12
Applying Data-in-place Analytics within Enterprise: IBM Open Data Analytics for z/os 13
IBM for z/os 14
Load balancer Intelligent Content routing Gateway / DataPower Security layer IBM z Analytics Enabling APIs for Data on IBM Z z/os Connect Enterprise Edition CICS Discovery Services IMS REST API Consumers Trusted zone WAS DMZ Outer APIs Inner APIs MQ REST Services DB2 Agile web, mobile, cloud access via REST to mainframe applications such as CICS, IMS, WAS
Load balancer Intelligent Content routing Gateway / DataPower Security layer IBM z Analytics Enabling APIs for Data on IBM Z z/os Connect Enterprise Edition CICS Data Virtualization Manager for z/os Adabas Discovery Services IMS IMS DB REST API Consumers Trusted zone WAS IDMS DMZ Outer APIs Inner APIs MQ SMF REST Services DB2 VSAM Agile web, mobile, cloud access via REST to mainframe data such as Adabas, IMS, VSAM DVM Non z/os
The Data Catalog Perspective
Map a Data Source to a create a Virtual Table
Automated Discovery & Configuration of data assets Multitude of Enterprise Applications & Data X X X X X Dataset names Cobol copybooks PL/1 include files IBM Application Discovery and Delivery 1 Intelligence (IBM 2 ADDI) IBM ADDI Catalog (Relational Store) IBM Open Data Analytics for z/os Apache Spark for Python / z/os Anaconda z/os z/os 4 Match high level language structures to physical datasets and databases Validate to associate type of data z/os Dataset Name Type COBOL Structure CSD.AI38.EMPLOYEE.KSDS VSAM EMPLOYEE-RECORD CSD.AI38.DEPARTMENT.KSDS VSAM DEPARTMENT-RECORD CSD.AI38.OFFICE.KSDS VSAM OFFICE-RECORD CSD.AI38.UPDATE.TRANS QSAM NAME-CHANGE-RECORD CSD.AI38.UPDATE.TRANS QSAM ADDR-CHANGE-RECORD CSD.AI38.UPDATE.TRANS QSAM SALARY-CHANGE-RECORD CSD.AI38.UPDATE.TRANS QSAM OFFICE-CHANGE-RECORD 3 5 Map Selected Generate virtual tables using simple wizards and batch processes with no additional data collection Optimized Data Layer 6 Read & analyze as result of SparkSQL or Python Interaction Cust_ID Investment Avg TX AMT Cust_ID Investment Avg TX AMT Analytic Output 1009530860 Cust_ID 114368 Investment 2090.32Avg TX AMT 1009530860 114368 2090.32 1009574010 112099 1297.41 1009530860 114368 2090.32 1009574010 112099 1297.41 1009578620 84638 1333.18 1009574010 112099 1297.41 1009578620 84638 1333.18 1009578620 84638 1333.18 Virtual Tables Consumed by IBM Open Data Analytics for z/os 7
Automated generation of code to access data from Spark & Python Reduce coding of data access via automated generation of Scala code (Python in progress) Automatically optimized based on available parallelism to generate best performance results without manual coding
More than just data access Access to all data from all applications without need to know which format the data it is in using the API of your choice MAINFRAME Adabas IMS DB Db2 for z/os IDMS SEQ files VSAM JDBC/ODBC IBM Data Virtualization Manager for z/os z/os Systems SMF records Log Strams NoSQL Virtual Tape zfs IBM z/os Connect EE Distributed Db2 Warehouse Db2 Warehouse on Cloud REST AP/SOAPI Db2 Informix
Selective direct Distributed Relational Database Architecture support Oracle Data Provider for DRDA (Linux) Oracle IBM Data Virtualization Manager for z/os DRDA Microsoft Service for DRDA SQL Server Apache Derby
MongoDB API for Mainframe Data Enables cloud/mobile developers to use mainframe data as if it was MongoDB MongoDB API for accessing: Adabas, Db2, IMS, IDMS, VSAM, Physical Sequential, IBM MQ. Accelerates time to value for mainframe participation in digital initiatives MongoDB API Socket Layer Message Processor Query Translator EXECSQL Mongo BSON Wire Protocol
MQ Queues and Messages Coming Soon SQL JSON Services IBM for z/os Eliminate the need to extract data from MQ MQGET Queue Manager MQPUT Non-relational Data Perform SQL-based analytics directly against MQ data z/os Adabas IDMS IMS VSAM SMF Join any DVM data sources with MQ messages
Where are opportunities for Data Virtualization? Digital applications needing access to Z data z/os Connect and APIs for web, mobile, cloud Any application that requires access to Z data Federation Server, DataStage, Optim, QualityStage, Cognos Hadoop Data Lakes, ETL, MongoDB Heritage Z data customers wanting to exploit their data in: VSAM, IDMS, IMS, ADABAS, Physical Sequential, SMF
Optimizing access to all Enterprise Data Simple Get transactional access, no data movement Open to all Apps Modern APIs enable access Secure Avoid risk by reducing moving data off Z Systems IBM Data Virtualization Manager for z/os Fast Exploits Z architecture, including parallelism and in-memory processing Cost Effective Keeps Z costs down with up to 99% ziip offload Non z/os data
Where to go for more information IBM Announcement http://www- 01.ibm.com/common/ssi/ShowDoc.wss?docURL=/co mmon/ssi/rep_ca/4/897/enus217-404/index.html&lang=en&request_locale=en YouTube "IBM for z/os" channel https://www.youtube.com/channel/uctbd_4ohohukdygssrl7sa Be sure to subscribe for new videos and to hit Like button Contacts: Steve Mink, mink@us.ibm.com Doug Dailey, douglasd@us.ibm.com Blanca Borden, blanca@us.ibm.com
28