soliddb Accélérer jusqu'à 10 fois l'accès à vos données grâce à IBM SolidDB, base de données en mémoire David Nightingale IT Specialist, IBM soliddb

Transcription

1 soliddb Accélérer jusqu'à 10 fois l'accès à vos données grâce à IBM SolidDB, base de données en mémoire David Nightingale IT Specialist, IBM soliddb

2 Disclaimer Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE. IBM, the IBM logo, ibm.com, IBM soliddb and IBM soliddb Universal Cache are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others. 2

3 IBM soliddb Market-leading In-Memory Relational Database Complements IBM s DB2/IDS as well as Oracle, Sybase and SQL Server with: micro-second response time high transaction throughput using the familiar SQL language % data availability IBM soliddb Universal Cache IBM soliddb Standalone 3

4 On-Disk VS In-Memory Databases On-Disk Databases All data stored on disk, disk I/O needed to move data into main memory when needed Traditional data structures like B-Trees designed to store tables and indices efficiently on disk Data is always persisted to disk Support very broad set of workloads, i.e. OLTP, data warehousing, mixed workloads, etc. Virtually unlimited database size In-Memory Databases All data stored in main memory, no need to perform disk I/O to query or update data Specialized data structures and index structures assume data is always in main memory Data is persistent or volatile depending on the in-memory database product Optimized for specialized workloads Database size limited by the amount of main memory Even when on-disk databases cache all data into main memory, in-memory databases provide shorter and more consistent response times and higher transaction throughput 4

5 The Solution: Relational, In-Memory, Database Technologies Process Performance-Critical Data 10 times faster Transactions per second Throughput of Tens of Thousands of Transactions per Second 160, , , ,000 80,000 60,000 40,000 20, Time In-memory database Disk-based database Microseconds Response Times Measured in Microseconds Select Update In-memory cache + disk-based database Disk-based database 5

6 IBM soliddb Features and Capabilities Flexible Data Access and Powerful SQL Programming Interfaces Java (JDBC 2.0/3.0, Hibernate, JBoss, WebSphere, Weblogic) C/C++ (ODBC 3.51, Solid API) Data Support All standard data types including CLOB/BLOBs Bulk import/export data tools Standard SQL ANSI SQL-92, SQL-99, SQL-2003 ACID transactional support Triggers, stored procedures, sequencers and events High Performance and Scalability Benchmarked to over 1 million tps SMP/Multi-core support Multi-user support In-memory terabyte data size capable Linked and Shared Memory Access models for application Persistence and Availability Persistence On disk (soliddb Standalone) On backend DB (soliddb Universal Cache) Availability Active HotStandby transparent architecture % availability Subsecond Failover Read-load balancing 6

7 IBM soliddb Features and Capabilities Ease of Use Very simple to install, deploy and maintain Full Installation uses only 50MB disk space Efficient use of system resources Designed to run unattended 365x24x7 No DBA required for day-to-day operations Robustness Very mature software technology Built to run in stringent, tough environments Able to handle software and hardware failures Data protection and integrity Proven and Widely Deployed History In production since 1993 More than 3 million deployments worldwide 100s of customers Many mission-critical real-time applications Telecom & Network Infrastructure Nokia-Siemens Networks, CISCO, HP, Nortel Networks, Telia-Sonera, NEC, Banking/Insurance Bank of America, GAD (Germany), Bank of New York Mellon, Retail Media Saturn 7

8 soliddb Combines Extreme Speed with Extreme Availability lication soliddb ODBC, JDBC drivers Two-node, hot-standby configuration for sub-second, transparent failover Automatic load-balancing of reads Configurable replication and logging settings to achieve right trade-off between performance, MTTR, and durability Subsecond failover Active database (primary) soliddb Every transaction gets replicated in real time Standby database (secondary) Standby database (secondary) 8

9 IBM soliddb lication soliddb ODBC, JDBC drivers Extreme Speed Keeps data in main memory at all times. Uses data structures and access methods specifically designed for storing, searching, and processing data in main memory with extreme speed Extreme Availability Supports % high availability Provides instant application failover and transparency to users Subsecond failover Low Cost Avoids costs of outages Ability to run virtually unattended Active database (primary) soliddb Every transaction gets replicated in real time Standby database (secondary) Sustains high throughput workloads with less hardware than disk-based databases 9

10 soliddb Universal Cache The industry s first relational, in-memory caching technology that accelerates IBM, Microsoft, Oracle and Sybase databases up to 10 times Universal Cache Tens of thousands of transactions per second Response times measured in microseconds 10

11 IBM soliddb Universal Cache Has Two Key Components 1. Relational, in-memory cache that delivers data with extreme speed Universal Cache 2. High performance data synchronization software that copies data back and forth between the in-memory cache and a specified backend database, making soliddb Universal Cache adaptable to different application and deployment needs Includes software to communicate with each supported backend database and with in-memory cache Includes tools for configuration and monitoring of cache synchonization 11

12 How IBM soliddb Universal Cache Works 2 soliddb Universal Cache loads performancecritical data from backend database 3 lications may connect to both in-memory cache and backend database to access and modify data 1 Administrator identifies performancecritical data and configures the cache Universal Cache 4 soliddb Universal Cache copies data back and forth between cache and a specified backend database 12

13 Adaptability: Graphical Tools For Schema Mapping and Data Transformations Universal Cache 13

14 Adaptability: Graphical Tools For Monitoring Cache Synchronization Universal Cache Monitor cache synchronization latency and throughput: 14

15 IBM soliddb Universal Cache: Supported Platforms and Backend Databases Supported platforms for the in-memory cache: AIX, HP-UX, Linux, Solaris, Windows Universal Cache Supported backend databases: DB2 LUW (V9.1, V9.5) on AIX, Windows, Solaris, Linux, zlinux DB2 z (V7, V8, V9) on z/os DB2 i (v5r4, v6r1 ) on i/os Oracle (9i, 10g, 11g) on AIX, HP-UX, Solaris, Linux, zlinux, Windows IDS V on AIX, HP-UX, Solaris, Linux, Windows Sybase (V12.5.4, V15) on AIX, HP-UX, Solaris, Linux, Windows Microsoft SQL Server (2000, 2005, 2008) on Windows 15

16 soliddb Use Cases

17 soliddb Managing 37+ Million Mobile Phone Subscribers Voice related services (i.e. voice mail) MRS SIP PROV MR Standby Server HLR Routing: Specifies redirection destination of the SIP application server, such as the voice mail system, from subscriber s telephone number and sends the information of the redirection destination to the terminal devices (i.e. mobile phone) by using SIP SIP Gateway soliddbtells SIP module which MRS to connect Primary Server soliddb UPS in City 1 KEY PROV: Provisioning MR: Media Resource MRS: Media Resource Server UPS: User Provisioning Server City 2 City n UPS 2 UPS n User Provisioning: After receiving new subscriber information from the HLR, soliddb provisions subscriber information to appropriate SIP application server 37 million subscribers (in each city) 20 microsecond SELECTs 24x7x365 operation with no database restart since go-live more than two years ago Number Portability: Makes it possible for subscribers to move from a phone service provided over the PSTN to one provided over the IP network, without the subscriber changing the phone number of the mobile phone 17

18 Financial Services Extreme Speed Use Case Enterprise Service Bus/Message Queue 1. Track realtime data changes 2. Process events/rules 3. Trigger events/alerts to notify other applications lication Universal Cache Business Challenge: Financial services news feeds, trades and settlements, equity positions, derivatives or cash market Access real-time market surveillance and trend analysis output Aggregate views of risk and suspicious activity for regulatory compliance and operational risk management Simulate auto-trading strategies Determine whether to place an order Example: Evaluate 30,000+ rules on 500 trades per second for 15 million users per day Technology Requirements Microsecond response times 10s of thousands to over 100K transactions per second Data persistence On-disk backend database for archival, reporting, analytics, etc. 18

19 soliddb supporting Session Management Session Management lication 1. Insert row when session starts 2. Perform single row selects and single row updates as session progresses 3. Delete row when session is completed Technology Challenges: Establish unique session state information at extreme speed for high numbers of concurrent users Maintain session state information for the duration of all active sessions, even in the case of system failures soliddb Benefits: - Extreme speed enabling real-time service delivery - Extreme data availability ( %) using a twonode, hot-standby configuration - Lower costs as a result of reduced CPU usage, and ability to run virtually unattended Session Data: Userid & Password Encryption keys Timestamp of session start Timestamp of last activity Session state information 19

20 soliddb supporting Session Management (continued) Session Management lication 1. Insert row when session starts 2. Perform single row selects and single row updates as session progresses 3. Delete row when session is completed Example: Session Management lication Each row (instance of session data) is small Few to 10s of KBs Each row is short-lived High Availability required Simple queries requiring extreme speed: One insert and one delete Single row selects and updates Session Data: Userid & Password Encryption keys Timestamp of session start Timestamp of last activity Session state information soliddb Benefits: - Access session information at extreme speed % availability with soliddb HotStandBy feature - Reduce CPU usage 20

21 soliddb supporting Data Synchronization Insurance lications soliddb Advanced Replication is used to synchronize data between sometimes-connected agents and central data stores Universal Cache soliddb Universal Cache communicates to multiple central databases and performs continual (or on-demand) replication making information required by agents available in one spot Technology Challenges: - Allow sometimes connected agents to access data from multiple central databases at extreme speed soliddb Benefits: - Advanced Replication feature allows for seamless synchronization of data for agents - soliddb Universal Cache communicates to multiple central databases - Access data at extreme speed 21

22 Learn more: THANK YOU! 22

23 soliddb Backup

24 Relational Database Software Powers Enterprise lications ERP CRM Data Warehousing General Ledger, Cash Management, Accounts Payable, Accounts Receivable, Fixed Assets, Human Resources, Payroll Sales and Marketing, Commissions Service Customer Contact and Call Center support Canned reports Ad-hoc Reporting OLAP Data Mining Leading Relational Databases Efficiently Support 100s to 1,000s of users Milliseconds to seconds response times 1,000s of transactions per minute 24

25 As Number of Users Increase and Data Volumes Grow Data Management Performance Must Increase 10x Communications Financial Services Web 2.0 Online Charging Authenticate and authorize Initiate service Manage credit balance Manage volume discounts Brokerage lication Receive market feed Evaluate equity positions Check for fraud Online Retail Web Site Authenticate user Manage personal wishlists Generate page contents with cross-sell data 100,000s to 1,000,000s of concurrent requests 10s of microseconds for database calls Evaluate 30,000+ rules on 500 trades per second for 15 million trades per day Facebook: 10,000,000 concurrent sessions = two billion page views a day Wikipedia: 3000 page views a second and 25,000 SQL requests per second 25

26 Adaptability: Data Synchronization Software - 1 High performance data synchronization software copies data back and forth between the in-memory cache and the backend database, making soliddb Universal Cache adaptable to different application and deployment needs Universal Cache Adapts to different application and deployment needs: 1. Read-only cache, data owned by the backend database 2. Read-write cache, data owned by the cache 3. Read-write cache, data ownership is shared Provides powerful schema mapping and data transformations, with graphical tools for configuration and tuning 26

27 Adaptability: Data Synchronization Software 2 How Data Synchronization Works - Overview Graphical tools to configure and monitor data synchronization management console Universal Cache soliddb agent backend (*) database agent Agents access cache or backend database to read and write data Agents communicate with each other to synchronize data between cache and backend database (*) soliddb Universal Cache includes agents for each supported backend database 27

28 Adaptability: Data Synchronization Software - 3 Read-Only In-Memory Cache, Data Owned By Backend Database lications access cached data for short response times and high throughput Data can only be modified in the backend database Changes made in the backend database can be synchronized to the in-memory cache, transaction by transaction, automatically or on-demand Universal Cache Ideal for applications that require fast access to data that changes occasionally, price lists, reference or lookup data to accelerate ETL (extraction, transformation, loading) when incrementally loading data warehouses, etc. 28

29 Adaptability: Data Synchronization Software - 4 Read-Write In-Memory Cache, Data Owned By In-Memory Cache lications can read, add, modify, or delete data only in the in-memory cache, but not in the backend database Changes are propagated from the in-memory cache to the backend database, transaction by transaction, automatically, or on-demand Backend database used for archival and reporting Universal Cache Useful for workloads that are read intensive with write transactions that have very stringent performance requirements that cannot be met with disk-based databases 29

30 Adaptability : Data Synchronization Software - 5 Example For Read-Write In-Memory Cache, Data Owned By the Cache Credit card application 10,000s tps INSERTs 10,000s tps INSERTs Universal Cache Fraud detection application 100,000+ tps SELECTs 10,000+ tps UPDATEs 100GB customer data (cache owned) Only current credit card transactions 100TB credit card transactions What does the application do: Fraud detection application correlates current credit card transactions with customer data for real-time analysis of fraud patterns and cancellation of identified fraudulent transactions before they complete How does it work: Cache owns all customer balance, address, shopping history data Cache temporarily stores current credit card transaction data for the fraud detection application to correlate with customer data Backend database stores history of credit card transactions Why soliddb Universal Cache: Fraud detection application needs to confirm or reject credit card transaction in real-time Requires response times of 10s of microseconds to perform analysis 30

31 Adaptability: Data Synchronization Software - 6 Read-Write In-Memory Cache, Data Ownership Is Shared lications can update the same data in both the inmemory cache as well as in the backend database at the same time Changes to the data can be automatically copied back and forth between the cache and the specified backend database Universal Cache Conflicts are detected and resolved by using pre-defined conflict resolution methods Useful when applications need to update the data in the backend database while the data is also cached for readwrite access 31

32 Adaptability: Data Synchronization Software - 7 Example For Read-Write In-Memory Cache, Data Ownership Is Shared Online charging application 100,000s tps Back-office subscriber application Subscriber self-service application Universal Cache Example: Communications application Online charging application correlates subscriber data with usage, roaming, discounts, etc. to calculate and apply the appropriate rate in real time Backoffice subscriber application provisions new mobile phone subscribers and services in the backend database Subscribers with pre-paid plans add minutes from a self-service application in the inmemory cache and expect immediate credit Subscriber data needs to be kept synchronized between the in-memory cache and the backend database Online charging applications requires response times of 10s of microseconds to calculate and apply the charge in real-time 32

33 Adaptability: Data Synchronization Software - 8 Schema Mapping and Data Transformations When configuring soliddb Universal Cache data schema, users have flexibility in deciding what data to cache: Retain the same relational database schema as the backend database, or have a different schema Universal Cache Load the entire database or select just specific tables, columns, and/or rows Extend the in-memory cache schema with additional tables or derived fields, and even complement them with stored procedures and triggers Maintain the data format or apply data transformations such as codepage conversions, data type conversions, custom data transformations, or summarizations 33

34 Adaptability: Data Synchronization Software - 9 Schema Mapping & Data Transformations Example: e-commerce Products (read-only in cache) Personalization (wishlists) Session management Universal Cache Data read only in the cache Filter rows: Products viewed most often Filter columns: Only those columns required to enable fast search Updateable in the cache and backend, bi-directional synchronization with conflict resolution No filtering Cache-only table for session management, user authentication, Products Personalization (wishlists) 34

35 Adaptability: IBM soliddb Universal Cache Scales Vertically soliddb Universal Cache scales on multi-core and multi-processor configurations Transactions per Second 100,000 80,000 60,000 40,000 20, ,929 61,352 39,489 13, Number of Processor Cores 35

36 Adaptability: IBM soliddb Universal Cache Scales Horizontally Example Stateless application Customer ID 1 to 400,000 Customer ID 1 to 400,000 Customer ID 1 to 400,000 Customer ID 1 to 400,000 lication uses load balancer to distribute workload across instances of soliddb Universal Cache Each instance of soliddb Universal Cache has the same data Updates to data in one of the inmemory cache instances are transparently propagated to the back-end database and then propagated to the other instances by the back-end database 36

37 Adaptability: IBM soliddb Universal Cache Scales Horizontally Customer ID 1 to 100,000 Customer ID 100,001 to 200,000 Customer 200, ,000 Customer 300, ,000 Large table is partitioned across instances of soliddb Universal Cache lication access data in inmemory cache instance soliddb Universal Cache synchronization capability maintains in-memory cache instances and backend database data consistent 37

38 4. Real-Time Data Changes soliddb 1. Tracks real-time data changes - 2. Processes events/rules 3. Triggers events/alerts to notify other applications On disk database: Customers Trade history Close history Enterprise Service Bus/Message Queue lications Universal Cache Financial services news, trades, settlements, etc E.g.: Evaluate 30,000+ rules on 500 trades per second Session Control: microsecond response time and extremely fast HA failover Data access patterns change frequently based on new rules Views of the data require table updates and occasional new tables Consistent high speed response time required When flexible and fast-changing data manipulation is required, soliddb is the best fit. Its SQL capabilities such as stored procedures make it easy to query and manipulate data. 38

39 soliddb Can handle large databases in memory Greater than Terabyte size in-memory Databases In 1Q we benchmarked a 1.6TB (Terabyte) in-memory relational database Insert throughput of 140,000 rows/sec Read throughput of over 540,000 queries/sec 1.7 million queries/sec achieved with client and server linked together Read response time of ~100 microsec 39

40 Launch flash demo from here (approx. 7 minutes if you watch the overview and all 4 use cases) d/streamed/ibm_demo_ibm_soliddb_universal_c ache-1-jan09.html 40

41 soliddb and soliddb Universal Cache Delivering Data at Extreme Speed IBM Confidential until announcement V6.3 Q V6.5 Accelerate not only DB2 and IDS but also Microsoft, Sybase and Oracle GUI Administration tools Globalization multi-byte character set support Increase write performance SQL Pass-though Shared memory access Data Aging Improved Cache performance Cache HA failover resilience Referential Integrity Improvements soliddb v6.5 Launch Globalization for AP Universal Cache All statements regarding IBM s future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. 41