Drilling Through The Stack. Burkhard Neidecker-Lutz Technical Director, SAP Research SAP AG

Transcription

1 Drilling Through The Stack Burkhard Neidecker-Lutz Technical Director, SAP Research SAP AG

2 Talk Outline Where I come from Layered stacks Violating layers for fun and profit The mother of all stacks: Future Internet SAP 2008 / Page 2

3 SAP Business Suite 12+ million users 121,000 installations 75,000 customers 140 countries 26 industry-specific business solutions 1 Software stack SAP 2008 / Page 3

4 The kinds of problems we handle Business Intelligence > Analytics over > ~50 TB data in memory with BIA Human Capital Management Payroll calculations for 500,000 employees in 3 hours Consumer Products 1.4 million sales order line items per day Portal 300,000 users Banking 40 million customers up to 8 million transactions an hour CRM SAP Business Suite SRM SCM PLM ERP Supply Chain Management 4.5M characteristic combinations & 512 GB - 1TB memory in live cache Utilities 25 million business partners 85 million service and sales orders per year Enterprise Resource Planning A customer with 5 users on a laptop Engineering & Construction 5,000 concurrent active users

5 Lots of data One size doesn t fit all Different types of data impose different requirements & different optimizations Progression Over Time Transactional Data Analytical Data Master Data Event Data Textual and Unstructured Data Order ~ 100G Write > read Many changes Accurate Consistent Performance Order > Tb Read only Slow changes Many queries Flexibility Performance Order ~ 1G Mostly read Mid change Many queries Distributed Order < Tb Many writes Few queries Distributed Filtering Correlation Order > Tb Mostly read Slow change Many queries Unstructured Contextual

6 What Do I Mean By Stacks Where I come from Layered stacks Violating layers for fun and profit The mother of all stacks: Future Internet SAP 2008 / Page 6

7 Example: Single machine stack Application App Server Libraries Operating System Processor/Memory File System or Database Storage Controller Block level storage

8 Why Layers Good engineering practice Hides complexity Allows interchangeability Allows for virtualization SAP 2008 / Page 8

9 Why break layers Some concerns span layers Non-functional properties Layer contracts can become obsolete over time Virtualization Shift of responsibilities Layering can hide important things Error handling Optimization opportunities SAP 2008 / Page 9

10 Digression: SAP Landscape in 30 seconds SAP XI SAP MDM SAP WAS SAP J2EE SAP BIA SAP Landscape RDBMS

11 Machine Stack, more realistically Application App Server Libraries Operating System RDBMS Libraries Operating System Filer OS Storage Controller Disk storage

12 Talk Overview Where I come from Layered stacks Violating layers for fun and profit The mother of all stacks: Future Internet SAP 2008 / Page 12

13 Playing With The Stack Application App Server Libraries Operating System RDBMS Libraries Operating System OSD OS Storage Controller Block Disk level storage storage

14 Introducing New Hardware Cache/Registers Cache/Registers Main Memory GDDR-RAM Main Memory Magnetic Disk Solid State Memory Tape/Optical Magnetic Disk

15 The common but stupid layering Application App Server Libraries Operating System RDBMS Libraries Operating System OSD OS Storage Controller SSD storage Disk storage

16 The interesting shift of layers Application App Server SSD storage Libraries Operating System RDBMS SSD storage Libraries Operating System OSD OS SSD storage Storage Controller Block Disk level storage storage

17 Eliminating layers Application App Server SSD storage Libraries Operating System RDBMS SSD storage Libraries Operating System OSD OS SSD storage Storage Controller Block Disk level storage storage

18 Transactions and Parallelism Database in memory Singlevolume logging contention Log buffer Parallel logging Log disk Log disk Log disk Log disk Serial restart 1. Checkpoint DB loading 2. Serial log replay Parallel restart CP disk CP disk CKPT disk Log disk CP disk CP disk CKPT disk Log disk Log disk Log disk

19 Textbook Transactions Image of a record 0000 Trans T1 Transactional log generation Trans T Log Trans T Write before-update image and afterupdate image ordered into log 0000 Replay L Replay L2 Replay L Log Recovery by log replay 1110

20 Eliminate Ordering Constraint Image of a record Differential log generation Trans T1 Trans T2 Trans T L1 L2 L Out-of-order log distribution L1 Log #1 L3 L2 Log #2 Out-of-order log replay 1110 Differential log image is set to XOR result between before-update image and afterupdate image Replay L1 L Replay L2 L Replay L2 L Replay L3 L Replay L3 L Replay L1 L Regardless of the log replay order, the same consistent record image is recovered

21 Results: new DBMS vs traditional DBMS Transaction per second Pure indexed search Processor 700 Mhz Xeon, 4 log disks Non-key field update Index field update Mixed (60% search 40% update) Insert Delete Response time (ms) ,5 10,8 42 4,8 4,8 4, ,8 Pure indexed search Non-key field update Index field update Mixed (60% search 40% update) Insert Delete 1416

22 Talk Overview Where I come from Layered stacks Violating layers for fun and profit The mother of all stacks: Future Internet SAP 2008 / Page 22

23 Future Internet: Service Webs FutureCar.com SmartBuilding.com egovbroker Custom-car design Car Connector Smart business start-up egovconnect securebanking Debt-wealth offsetting Entrepreneur.com MediaMart.com NewsTopia TelcoConnect Service Delivery Platform Service Channel Service Broker Service Aggregator Business Model (e.g. Marketplace) Business Model (e.g. Marketplace) Service Gateway Service Provider SOA Order CRM Payments Business management SCM Analytics partner mgmt Warehouse management Service deployment & registration Service composition & mediation SLA & contract management Service presentation management Search/ discovery Authentication & authorization Systems monitoring & management SAP 2007 / Page 23 Business Process Platform

24 Chasing nonfunctional properties through the stacks General Process Model business processes, variants, steps mapping to application components Default nonfunctional requirements Business Configuration & SLAs Customized Process Model customized processes & components & non-functional requirements Functionally complete solution spec. Process Performance Engineering Application Component Model Unbound Model abstract landscape capacity constraints (Virtualized) infrastructure models Grounded Model resource/ virtualization choices Complete functional & infrastructre spec. Deployment & Virtualization Performance Engineering Available resources Bound Model virtual > physical mapping Physical resources allocated No downtime (e.g. migrate VM) Deployment service Deployed Instance services available to users Operational management Customer Service provider Software provider Infrastructure provider Packaging Model Execution Component Model May cause downtime (change from virtual to Physical Machine) Behaviour Prediction Service Layered Queueing Models; trace based performance prediction etc Operation data obtained through monitoring