Fast, In-Memory Analytics on PPDM. Calgary 2016

Transcription

1 Fast, In-Memory Analytics on PPDM Calgary 2016

2 In-Memory Analytics A BI methodology to solve complex and timesensitive business scenarios by using system memory as opposed to physical disk, by increasing speed, performance and reliability when querying relevant data * milliseconds nanoseconds Spinning HDD SSD NVME SSD DRAM CPU Memory * Webopedia 2

3 Why In-Memory Increased Speed Decreased Costs 64 bit Architecture Large addressable memory Many options Oracle DB, MSFT SQL Server, SAP Hana VoltDB, Memsql, Spark, several others Speed Cost 3

4 Most Common PPDM Implementations PPDM DB on Oracle SQL Server PostgreSQL Virtual or dedicated DB Server Primarily on-disk Sources Staging PPDM Target 4

5 In-Memory PPDM Implementation Staging PPDM Sources Target PPDM Database Staging and/or PPDM Full / partial in-memory Durable / temporary HDD / SSD RAM 5

6 SQL Server 2016 : In-Memory SQL SERVER 2014 SQL SERVER 2016 In-Memory DB Size Schema Execution plan MAX of 256 Gb No schema changes allowed after creation No parallel plans All available memory Schema can be altered after creation Certain parallel plans allowed TSQL and other SQL constructs Always-On Memory Management Other features Limited SQL constructs Delayed transactions for Always-On Basic garbage collection Much more expanded set Concurrent transactions on Always-On Improved garbage collection Several other technical improvements 6

7 A Few Caveats (SQL Server specific) Data compression, bulk logging, linked servers, replication, mirroring, DDL triggers and some others not supported Some Transact SQL features not supported for in-memory tables (merge into clause, etc.) 7

8 In-Memory Architecture Guiding Principles Be selective on in-memory tables / data Look for performance boosting operations Leverage fast, concurrent data writes Data manipulations via native procedures Indexes :- Hash for unique data or data with few duplicates Non-clustered for data with large set of duplicates Be self contained 8

9 Implementation Procedure Identify primary and associated data sets Modify PPDM DDLs for in-memory tables Convert clustered indexes to non-clustered (existing db.) Selectively exclude indexes on in-memory tables Drop foreign keys constraints referencing unique keys constraints Estimate memory needs and allocate physical space Instantiate DB with in-memory file group Load data 9

10 Our Implementation PPDM Bit Windows VM SQL Server Core 2.4 GHz Xeon 64Gb DDR4 RAM 250Gb Samsung SSD 1TB WD HDD Frequently used tables converted to in-memory Three types of test instances HDD, SSD, In-Memory 10

11 Performance Metrics In-memory tables: Well tables Production tables Associated reference tables 1M simulated Well data 10M simulated Prod data No additional indexes Lower is better In-Memory operations are consistently faster 11

12 Our Recommendations Start small with a targeted data set Minimize indexes on memory optimized tables Leverage in-memory tables for frequently used data Use Schema_Only option for staging/ temporary data Choose Native SPs over Interpreted SQL where possible Make data store self contained 12

13 Summary In-memory tables Provide an order of magnitude performance gains Are cost effective Lower tuning & maintenance efforts Gain competitive advantage 13

14 Questions, Additional Information Gautam Pamula