zspotlight: Spark on z/os

Transcription

1 zspotlight: Spark on z/os Avijit Chatterjee, STSM, IBM Competitive Project Office 1

2 CEOs are increasingly focused on customers as individuals leveraging contextual mobile and cognitive technologies 2015 IBM Global C-Suite Study: The CEO Perspective 2

3 To focus on customers as individuals, real-time analytics is heavily leveraged across many industries Banking Card: Use scoring to determine transaction risk based on spending history Money laundering risk: Based on money wiring to multiple accounts keeping amount below threshold Retail Sales opportunity: Real-time scoring for target marketing 3 Government Compliance: Score to detect non-compliant behavior and tax evasion Social Services: Assess likelihood that individual will need multiple agency support to proactively engage various agencies to create best outcome and manage costs

4 Running Analytics on enterprise data off-platform doesn t pay for a mainframe-centric business Analytical Analytical Analytical A large European bank: 120 database images created from bulk data transfers 1,000 applications on 750 cores with 14,000 software titles ETL consuming 28% of total distributed cores and 16% of total MIPS A large Asian bank: Source: IBM Eagle Studies Operational Analytical Analytical One mainframe devoted exclusively to bulk data transfers ETL consuming 8% of total distributed core and 18% of total MIPS With this strategy, IT costs grow faster than business growth 4

5 Rather it leads to significant data transfer costs 5 Example: Source: IBM CPO internal study Operational x86 1 TB of data transferred per day ODS Analytical Analytical Analytical ETL Calculator: Estimated 4 yr. cost summary System costs = $9,864,412 Labor costs = $393,927 Total = $10,258,339 Assuming 4 cores on z13 running at 85% utilization and 12 cores on x86 servers run at 45% utilization, transfer will burn 519 MIPS and use 10 x86 cores per day

6 Apache Spark powers a federated Analytics architecture keeping data in place Unified analytics platform for structure and unstructured data Flexibility & agility with multi-language support Efficient structure 100x vs. map reduce Rich set of built-in functions with consistent APIs: Spark SQL, Spark MLib, Spark Streaming, GraphX, Spark Spark Spark DB2 Linux on z Systems CICS Spark VSAM IMS DB SMF Syslog IMS Spark Physical Sequential z/os Tape Spark WAS DB2 z/os Log Streams ADABAS Spark Spark Power Spark Spark x86 Spark Spark Cloud 6

7 Apache Spark empowers users to accelerate the insight economy What they want to do: Build applications that leverage advanced analytics in partnership with the data scientist and data engineer Follow agile design methodologies Optimize performance and meet SLAs Scientist the convincer What they want to do: Identify patterns, trends, risks, and opportunities in data Discover new actionable insights How Spark can help: Supports the entire data science workflow: from data access and integration, to machine learning, to visualization Provides a growing library of machine learning algorithms 7 How Spark can help: Supports the top analytics app languages such as Python and Scala Eliminates programming complexity with libraries such as MLlib and simplifies DevOps Makes it easy to embed advanced analytics into applications App Developer the builder Engineer the thinker What they want to do: Bridge between the Scientist and the App Developer Implement machine learning algorithms at scale Put the right data system to work for the job at hand How Spark can help: Abstract data access complexity (Spark doesn t care what your data store is) Enables solutions at web-scale

8 Apache Spark is an operating system for Analytics Spark Streaming: Enables scalable, high-throughput processing of live data streams Live stream chopped into batches based on time window Spark SQL: Provides capability to perform relational queries via SQL (subset of HiveQL) Mix SQL queries with Spark applications Spark MLIB Provides scalable machine learning library, has common machine learning functions Provides classification, regression, clustering, filtering, etc. from Spark GraphX Spark APIs for graph style processing and iterative graph computations Spark Core: Foundation providing task dispatching, scheduling, i/o Representation of Spark s basic unit of data: RDD 8

9 On-platform Operational Analytics using Spark on z/os z/os JDBC/AZK DB2 z/os NYSE data Spark on z/os joins multiple data types for fast, complete analytics, without moving the data 1 Scala query using sql syntax to access 3 data sources (over 1 billion rows of data) in 3 different formats JDBC/AZ K JDBC/AZ K Flat file Nasdaq data VSAM S&P data 1.1 billion rows of source data Filtered to 50M rows of Trades for brokers in 1 region Summarized to show activity for each broker across the big 3 exchanges Completed in < 2 minutes with 1 GP (60% utilized) and 11 ziips on a z13 LPAR with 512Gb memory Use Case: Filtered data pull from 1 Billion rows, using Spark filtering to access 50 Million rows, and then summarize using Spark aggregation 9 Resource specs for run: 6 executors, 6Gb driver mem, 80Gb per executor, 4 vcores per executor (ziips in SMT-2), 12 concurrent threads, 512Gb total memory in LPAR, 13 ziips, 2 GPs. AZK accessing all data, spark unionall to merge data. 55% of 1 GP utilized during peek resource consumption

10 On-platform Operational Analytics achieves 67% lower TCA CICS DB2 ETL Apache Spark Parquet $2,105,990 (3 yr. TCA) 10 Trade 166GB Brokerage aggregation query workload across Trades tables from 3 exchanges (over 5 Billion trades, 500GB) * 3-Year TCA includes 3-year US prices for Hardware, Software, Maintenance and Support as of 05/16/2016. Price and performance for x86 environment includes cost of ETL and elapsed time to transfer the data. This is based on an IBM internal study designed to replicate a typical IBM customer workload usage in the marketplace. z/os z Competitor x86 System Intel E v2 2.7GHz 12co CICS DB2 Apache Spark z/os z ziips Linux 67% $697,106 (3 yr. TCA) lower TCA* For systems compared

11 It doesn t pay to move data to x86 to run Analytics beyond 150GB Competitive Project Office $4,000,000 Comparing 3-year TCA factoring Cost of ETL $3,500,000 $3,000,000 $2,500,000 $2,000,000 $1,500,000 $1,000,000 $500,000 $0 100GB 300GB 500GB 700GB 900GB Spark on z/os Spark on x86 11

12 Spark on z/os offering is now generally available What is in the Offering? IBM z/os Platform for Apache Spark (IBM product): Apache Spark enabled for z/os Optimized Integration Layer No License Charge product Support & Service available from IBM Very aggressive pricing for ziips and memory for Spark z/os workload Lab Services install, config, tune Jumpstart service for data science use cases FactZ quick strike POCs for Spark analytic business applications Ecosystem GitHub z/os-spark repository Jupyter Notebook IDEs (Scala Workbench, Interactive Insights Workbench) Apache Job Server Sample data & code snippets Rocket: Industry vertical mappings, e.g. ISO for card data In progress: R support FactZ: Custom Solutions for banking & insurance Zementis: Fast, Scalable, In-Transaction Predictive Scoring integration Apache Spark 12

13 Common use cases of Spark on z/os Analytics across OLTP & Warehouse information Analytics combining business-owned data and external IoT and social data Analytics to improve system performance and operations in realtime using streaming as well as archived data 13

14 Why Spark on z/os is best fit Optimized and parallel access to almost all z/os data environments and distributed data sources analyzing data in place Spark memory structures with sensitive data are governed with z/os security capabilities Leverages z/os memory management, compression, and RDMA communications to provide a high-performance scale up and scale out architecture. Uses large pages, incorporating DRAM with large amounts of Flash as an attractive means to provide scalable elastic memory Best fit analytic capability for the investments made in SMF in-memory analytics Leverages zedc compression when compressing internal data for caching and shuffling SMT2 and SIMD on select operations for enhanced performance Very high ziip offload -- for affordability Intra-SQL and intra-partition parallelism for optimal data access 14

15 Seize the moment with Analytics on z Systems adopt or be disrupted Enhance customer experience by providing right product at the right place and right time Locate analytics exactly where your data resides Power real time analytics with in-memory analytics engine Spark on z/os Get started today: ibm.biz/zanalytics 15

16 IBM z Systems CPO Resources IBM Competitive Project Office Our mission is to perform hands-on competitive research that will enable sales and technical professionals to boost market awareness, increase pipeline and sales of IBM z Systems hardware and IBM software. Please visit our website for additional information on how we can help you generate leads and close business. Customer Briefing Bringing to life the results of IBM's lab-based, hands-on competitive research with your clients in face-to-face events. "IBM z Systems More Potential than Ever" "IBM LinuxONE - Enabling IT to Solve Business Challenges" For First in Enterprise (FIE) IBM CPO Competitive Sales Assists (CSA) A CPO CSA helps to make the advantages of IBM's solutions more real and compelling to a customer evaluating competitive alternatives. Visit our website to learn how CPO provides one-to-one clients engagements to assist sellers with closing competitive opportunities. IBM CPO Communication Share the external z Systems website with your clients. Click here to subscribe to IBM CPO Communications IBM Redbooks Point-of-View and Redpaper publications Authored by IBM CPO Read more about our research - CPO Technical Deliverables Join the CPO Community 16 For information on z Systems CPO Program or Events: Sally Touscany touscany@us.ibm.com