Application-Tier In-Memory Analytics Best Practices and Use Cases

Application-Tier In-Memory Analytics Best Practices and Use Cases Susan Cheung Vice President Product Management Oracle, Server Technologies Oct 01, 2014 Guest Speaker: Kiran Tailor Senior Oracle DBA and Architect Chartered Institute of Management Accountants

Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

Program Agenda 1 2 3 4 5 6 TimesTen In-Memory Database Overview TimesTen for In-Memory Analytics Common Use Cases and Best Practices Real-life Customer Use Case at CIMA Product roadmap Q&A

Oracle TimesTen In-Memory Database Microseconds Response Time in Application Tier Relational Database Pure in-memory ACID compliant Standard SQL Entire database in DRAM Extremely Fast Microseconds response time Very high throughput Persistent and Recoverable Database and Transaction logs persisted on disk and flash storage (local to TimesTen) Compatible with Oracle Database Data types, PL/SQL, OCI, ODP.NET, PHP, R Integrated with RAC, Data Guard, Enterprise Manager, SQL Developer, etc. 5

TimesTen In-Memory Database Persistence and Recovery TimesTen database persistence On local storage where TimesTen runs Flash, SSD, hard disk storage (HDD) TimesTen Checkpoint Files TimesTen Transaction Log Files All transactions are logged and persisted Redo, undo, and recovery Dual database Checkpoint files Database restart Database recovery after failures Separate storage for Transaction Logs and Checkpoint files if using HDD 6

Application Development with TimesTen SQL PL/SQL Oracle R ODP.NET ODBC OCI, Pro*C ttclasses Languages C/C++/C# Java/JDBC J2EE App Servers OR Mapping

TimesTen In-Memory Database Low Latency - Microseconds Response Time TPTBM Read and Update E7-4890 v2 @ 2.80GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen 11.2.2.7.5 (100M rows, 17GB) Microsecon nds 10.00 8.00 6.00 4.00 2.00 0.00 Response Time Millionths of a Second 3.58 SELECT Query Millionths of a Second 9.07 UPDATE Transaction

TimesTen In-Memory Database Read Scalability 25 Million Queries per Second 30,000,000 25 Million Queries per Second TPTBM 100% Read E7-4890 v2 @ 2.80GHz 4 sockets, 15 cores/socket, 2 threads/core TimesTen 11.2.2.7.5 (100M rows, 17GB) Queries Pe er Second 25,000,000 20,000,000 15,000,000 10,000,000 5,000,000 0 25,192,607 279,262 1 10 20 30 60 90 120 Concurrent Processes 9

TimesTen In-Memory Database 1.3 Million Update Transactions Per Second Per Processor 1,400,000 Update Throughput Per Processor 1.3 Million Update TPS TPTBM 100% Update E7-4890 v2 @ 2.80GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen 11.2.2.7.5 (100M rows, 17GB) Update Transaction Per Second 1,200,000 1,000,000 800,000 600,000 400,000 200,000 0 1 5 10 15 20 25 Concurrent Update Processes 10

TimesTen In-Memory Database 3.4 Million Transactions Per Second Per Processor 3,500,000 Mixed Workload Throughput Per Processor 3.4 Million Transactions Per Second TPTBM 100% Mixed Workload (80-10-5-5) E7-4890 v2 @ 2.80GHz 1 socket, 15 cores/socket, 2 threads/core TimesTen 11.2.2.7.5 (100M rows, 17GB) Transactions Per Second 3,000,000 2,500,000 2,000,000 1,500,000 1,000,000 500,000 0 1 5 10 15 20 25 Concurrent Processes 80-10-5-5 Workload = 80% select, 10% updates, 5% inserts, 5% deletes 11

Oracle TimesTen In-Memory Database Deployment Options 1. Standalone In-Memory Database for OLTP applications Application Standalone In-Memory DB for OLTP 2. In-Memory Cache Database for the Oracle Database 3. In-Memory Analytics deployed on Oracle Exalytics and commodity hardware Application In-Memory Analytics Commodity Hardware In-Memory OLTP Caching Oracle Exalytics

Oracle TimesTen Pure In-Memory Relational Database Very Fast Response Time for Very High Throughput in Application Tier 1996 2005 2006 2008 2009 2011 2012 2013 2014 Pre-Oracle acquisition 1998 First commercial In-Memory RDBMS HA Replication Online Upgrades Application-tier Cache for Oracle Database TimesTen 6 TimesTen 11g TimesTen 11g 11.2.2.x TimesTen 7 11.2.1 11.2.2 Enhancements Oracle RAC integration National Language Support Oracle Data Types support SQL Developer Integration Enterprise Manager integration PL/SQL and OCI Support Oracle Clusterware Integration Cache Grid for Scale Out ODP.NET Support BLOB, CLOB, NCLOB data types Parallel Replication In-Memory Analytics Columnar Compression Index Advisor Oracle R Support In-Memory Star Join Oracle Golden Gate Integration Parallel data import from Oracle Database Parallel database restart Highly concurrent range indexes

Analytics Use Cases for Oracle TimesTen

In-Memory Analytics - Summary Aggregates Scenarios Aggregations are suitable for analysis of higher-level grains of FACT data Knowledge of query patterns required Aggregated tables and indexes typically much smaller than detail-tables Reports using summary aggregates typically provide exceptional response time (sub-second) Operational Data Store 1 2 Reports Using Aggregate Set 1 OLTP Data Warehouse Summary Aggregate Tables and Indexes stored in TimesTen Reports Using Aggregate Set 2

In-Memory Analytics - Data Mart Use Cases Subset of Data from a Data Warehouse Data mart and operational data stores For analysis where summary aggregates are not sufficient Require access to detail source tables (fact and dimension tables) Consider hot set of data* Data volume constrained by availability of RAM in the system May use compression to include more data Common use cases have both aggregations and detail tables If the entire data warehouse is too big to fit into memory, only load portion of the data to TimesTen

In-Memory Analytics ODS and Data Mart Detail Tables Scenarios Operational Reports Operational Data Store OLTP Data Warehouse Aggregates + Detail tables + indexes Reports with Aggregates AND Detail Tables

Best Practices Tips and Tricks

TimesTen In-Memory Database Storage Speed Matters to Database Restart Time Storage speed matters Faster read operations using Flash and Solid State Disks Higher throughput with parallel threads Significant time reduction for database load to memory TimesTen Checkpoint Files 19

Dual Parallelism Oracle TimesTen and Intel NVMe P3700 Speed up Database Load Time to Memory 100GB Database Load to Memory Time in Seconds 700 600 500 400 300 200 100 0 Database Load Time 630 212 49 HDD 15K RPM Sata SSD Intel NVMe P3700 TimesTen 11.2.2.7 on Oracle Linux, Intel Xeon E5-2690 3.0 Ghz, 2 processors 20 core server 20

Dual Parallelism Oracle TimesTen and Intel NVMe P3700 Multiple Flash Cards Lower Load Time and Higher Throughput 60 50 40 30 20 10 0 100GB Database Load to Memory Time in Seconds 42 Single NVMe P3700 30 Striped Two P3700 100GB Database Load to Memory Read Rate Single NVMe P3700 Striped Two P3700 Cards 2.2 GB/s 3.3 GB/s 0 1 2 3 4 TimesTen 11.2.2.7 on Oracle Linux, Intel Xeon E7-4890 v2 2.8 Ghz, Two 15-core Processors 21

TimesTen Cost-Based SQL Optimizer Proper Indexes and Up-to-date Statistics Optimized Query Execution Plans Hash indexes Best performance for equality matches Must be properly sized for good performance Undersized hash indexes can result in severe performance penalties Range Indexes Range indexes best for range search Can ALTER Unique PK range indexes Default index type for primary key indexes to hash indexes Up-to-date table and column statistics Important for a cost-based SQL optimizer

Summary TimesTen is an excellent choice for in-memory analytics in the application tier Correct setup and usage is vital to achieve good performance and easy management OS, configuration, operations, persistence storage, tuning, etc. Use the tools available to make your life easier and get the best results Many more best practices available Visit TimesTen product center on Oracle Technology Network Refer to TimesTen documentation for more information (available online)

Most Widely Used Relational In-Memory Database Deployed by Thousands of Companies

Charter Institute of Management Accountants (CIMA) Real-life Use Case of In-Memory Analytics Kiran Tailor Senior Oracle DBA and Architect Chartered Institute of Management Accountants

CIMA, Exalytics and TimesTen Kiran Tailor Senior Oracle DBA and Architect The Chartered Institute of Management Accountants Date: 01/10/2014

About Me Kiran Tailor, Senior Oracle DBA/Architect Sixteen years working with Oracle Started working with Oracle Forms, moved to databases Oracle Databases for 15 years OBIEE for more than five years CIMA CIMA,Exalytics,TimesTen 01/10/2014 2

CIMA Our mission is to help people and businesses to succeed in the public and private sectors We have more than 218,000 members and students in 177 countries They work at the heart of business in industry, commerce and not for profit organisations We have 28 offices around the world, employing 450 people We constantly update our qualification. This ensures that our members and students are first choice for employers who are recruiting financially trained business leaders In 2012, together with the AICPA, we launched the CGMA designation CIMA CIMA,Exalytics,TimesTen 01/10/2014 3

Our Challenges Meaningful information from the systems, that we could action upon instantly CIMA CIMA,Exalytics,TimesTen 01/10/2014 4

Our Challenges As systems were slow. Staff/Business managers constantly phoning for information CIMA CIMA,Exalytics,TimesTen 01/10/2014 5

Our Challenges Marketing Campaign Reports 8 minutes Exam Reports 2 minutes individually, thousands to do Anything with Activities 15 minutes+ CIMA CIMA,Exalytics,TimesTen 01/10/2014 6

CIMA Before Exalytics DW OLTP Siebel CRM CIMA CIMA,Exalytics,TimesTen 01/10/2014 7

Exalytics/TimesTen common Setup DW OLTP Siebel CRM CIMA CIMA,Exalytics,TimesTen 01/10/2014 8

TimesTen CIMA Set Up Siebel DW CIMA CIMA,Exalytics,TimesTen 01/10/2014 9 Siebel CRM

TimesTen DAC CIMA Set Up How DAC identifies records in TimesTen CIMA CIMA,Exalytics,TimesTen 10

TimesTen DAC CIMA Set Up How DAC identifies records that need to be copied to TimesTen CIMA CIMA,Exalytics,TimesTen 01/10/2014

TimesTen DAC CIMA Set Up How DAC identifies records in TimesTen at task level CIMA CIMA,Exalytics,TimesTen 01/10/2014 12

TimesTen DAC CIMA Set Up How DAC identifies which records have changed in the warehouse that need to be copied to TimesTen CIMA CIMA,Exalytics,TimesTen Siebel CRM 01/10/2014 13

Outcome Full warehouse approx 300gb loaded into memory 100x + performance improvement, in some cases much higher Full load took approx 3 hours Incremental nightly copy approximately 6 minutes Achieved huge performance uplift with zero application code changes Enabled previously unachievable analysis, leading to greater knowledge of customers and challenges across the company 100% In-memory Daily load time 6 mins ipad apps roll-out ongoing Zero Application code changes Performance Improvement INTELLIGENCE AND BUSINESS RESULTS CIMA CIMA,Exalytics,TimesTen 2014 14

Feedback It s all a lot better. It s so quick now it s difficult to measure without an atomic stopwatch. UK User I didn t used to use the old dashboards to the extent that I do now- maybe that s partly because I used to get so fed up with the speed that I gave up whereas now it s there at the press of a button UK User Report UK - WAN connection on standard Analytics 228.6 seconds Outside of UK - Wifi connection on Analytics not possible due to error Outside of UK - Wifi connection on Times Ten In Memory DW 1.1 second via webex CIMA CIMA,Exalytics,TimesTen 2014 15

Deployment Tips In RPD make sure you use correct TimesTen ODBC settings We used DAC to create the table and index scripts INITIAL load: run the table scripts on the TimesTen Server, load the data using the DAC, run the create index scripts on the TimesTen server, update the stats on TimesTen OBIEE Caching: Linux Ram Disk cache into memory! If using Exalytics and LDAP for OBIEE, follow the weblogic admin guide for configuration Keep checking support for Patch updates If any columns are being used to do updates make sure they are indexed. Update the stats constantly especially if you are updating and inserting rows CIMA CIMA,Exalytics,TimesTen 01/10/2014 16

Dziękujemy! Thank you! Спасибо! 謝謝! ﺷﻛرا ﻟك Kiran Tailor kiran.tailor@cimaglobal.com

Q&A