status Emmanuel Cecchet

Transcription

1 status Emmanuel Cecchet JOnAS developer workshop - c-jdbc@objectweb.org 1-23/02/2004

2 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 2-23/02/2004

3 Motivations J2EE performance scalability bounded by database performance Database tier must be scalable fault tolerant (high availability + failover) without modifying the client application using open source databases on commodity hardware JOnAS developer workshop - c-jdbc@objectweb.org 3-23/02/2004

4 How do we want to use? end-to-end open source solution for large scale J2EE clustered application servers cost-effective solution for both small and large configurations Apache Internet MySQL JOnAS developer workshop - c-jdbc@objectweb.org 4-23/02/2004

5 Redundant Array of Inexepensive Databases RAIDb controller gives the view of a single database to the client balance the load on the database backends RAIDb levels RAIDb-0: full partitioning RAIDb-1: full mirroring RAIDb-2: partial replication composition possible JOnAS developer workshop - c-jdbc@objectweb.org 5-23/02/2004

6 C-JDBC Middleware implementing RAIDb Two components generic JDBC 2.0 driver (C-JDBC driver) C-JDBC Controller C-JDBC Controller provides performance scalability high availability failover caching, logging, monitoring, Supports heterogeneous databases JOnAS developer workshop - c-jdbc@objectweb.org 6-23/02/2004

7 JDBC Without C-JDBC Using JOnAS with C- JDBC Driver Database JOnAS developer workshop - c-jdbc@objectweb.org 7-23/02/2004

8 Using JOnAS with C- JDBC Update the datasource to use the C-JDBC driver JOnAS Driver Virtual Database Controller JDBC Driver Cache Pooling Backend JOnAS developer workshop - c-jdbc@objectweb.org 8-23/02/2004

9 What C-JDBC offers No application modification No JOnAS modification No database modification Virtual Virtual Database Database Controller Load balancing Driver JDBC Driver Cache Pooling Backend Backend recovery JOnAS developer workshop - c-jdbc@objectweb.org 9-23/02/2004

10 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 10-23/02/2004

11 Inside the Controller Java client Sockets RMI program (Servlet, EJB,...) C-JDBC driver Sockets RMI Administration console XML configuration file JMX C-JDBC Controller Configuration & administration XML engine Virtual database Authentication Manager Request Manager Virtual database Authentication Manager Request Manager Recovery Log Scheduler Request Cache Load balancer Recovery Log Scheduler Request Cache Load balancer Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager MySQL JDBC driver MySQL JDBC driver MySQL JDBC driver Oracle JDBC driver Oracle JDBC driver MySQL MySQL MySQL Oracle Oracle JOnAS developer workshop - c-jdbc@objectweb.org 11-23/02/2004

12 Fault tolerant recovery log Client application (Servlet, EJB,...) C-JDBC driver C-JDBC Controller Checkpointing service Recovery virtual database Application virtual database Authentication Manager Request Manager Authentication Manager Request Manager Scheduler Load balancer Recovery Log C-JDBC driver Scheduler Request Cache Load balancer Database Backend Database Backend Database Backend Database Backend Database Backend MySQL MySQL MySQL MySQL JOnAS developer workshop - c-jdbc@objectweb.org 12-23/02/2004

13 Cascading controller Client program C-JDBC driver JVM Client program C-JDBC driver JVM Client program C-JDBC driver JVM C-JDBC controller Full replication C-JDBC controller Full replication DB native JDBC driver C-JDBC driver C-JDBC driver DB 1 DB 2 C-JDBC controller Full replication DB native JDBC driver C-JDBC driver DB 3 DB 4 C-JDBC controller Full replication DB native JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org DB 5 DB 6 DB /02/2004

14 Heterogeneity support application already written for a specific [commercial] database user defined rules for on-the-fly query rewriting to execute on heterogeneous backends Java client program C-JDBC driver JVM Oracle JDBC driver Servlet container Tomcat, Jetty,... C-JDBC driver JVM C-JDBC Controller RAIDb-2 EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM MySQL JDBC driver Oracle MySQL MySQL MySQL JOnAS developer workshop - c-jdbc@objectweb.org 14-23/02/2004

15 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 15-23/02/2004

16 Query caching status Cache contains a list of SQL->ResultSet Policy defined by querypattern->policy 3 policies EagerCaching: variable granularities for invalidations RelaxedCaching: invalidations based on timeout NoCaching: never cached RUBiS bidding mix with 450 clients No cache Coherent cache Relaxed cache Throughput (rq/min) Avg response time 801 ms 284 ms 134 ms Database CPU load 100% 85% 20% C-JDBC CPU load - 15% 7% JOnAS developer workshop - c-jdbc@objectweb.org 16-23/02/2004

17 Throughput: 3900 pages/min RUBiS - Tomcat without C-JDBC caching 1 Database 100% cpu Tomcat ~50% cpu JOnAS developer workshop - c-jdbc@objectweb.org 17-23/02/2004

18 Throughput: 4200 pages/min RUBiS - Tomcat with C-JDBC relaxed caching Tomcat ~55% cpu 1 Database ~20% cpu C-JDBC <10% cpu JOnAS developer workshop - c-jdbc@objectweb.org 18-23/02/2004

19 JOnAS and C-JDBC query RUBiS caching Throughput (req/min) No cache Response time (ms) No cache Servlet JOnAS SF CMP JOnAS CMP JOnAS developer workshop - c-jdbc@objectweb.org 19-23/02/2004

20 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 20-23/02/2004

21 Horizontal scalability JGroups for controller synchronization Groups messages for writes only Client 1 Client... Client n Client 1 Client... Client n Controller1 Controller1 Controller2 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 JOnAS developer workshop - c-jdbc@objectweb.org 21-23/02/2004

22 Horizontal scalability Centralized write approach issues commit commit Controller1 Controller2 Controller1 Controller2 Controller2 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 commit commit???? Issues with transactions assigned to connections T1(Read) T1(Write) Controller1 Controller2 Controller1 Controller2 Controller1 Controller2 DB1 DB2 DB1 DB2 DB1 DB2 T1(R) T1(W) T1(R) T2(W) JOnAS developer workshop - c-jdbc@objectweb.org 22-23/02/2004

23 Horizontal scalability General case for a write query 3 multicast + 2n unicast DistributedVirtualDatabase@controller1 execwriterequest(abstractwriterequest) return result DistributedRequestManager Scheduler Distributed scheduling handler Write query handler Send request to all controllers Wait for schedulers replies DistributedVirtualDatabase@controller2 DistributedRequestManager Distributed scheduling handler Distributed load balancer handler Notify scheduler of completion Scheduler Load Balancer Load Balancer Distributed load balancer handler Notify scheduler of completion Send execute request order to all controllers Wait for load balancer replies Notify controllers of completion success or not DistributedVirtualDatabase@controller3 DistributedRequestManager Distributed scheduling handler Scheduler Return result Distributed load balancer handler Load Balancer Notify scheduler of completion JOnAS developer workshop - c-jdbc@objectweb.org 23-23/02/2004

24 Horizontal scalability Solution: No backend sharing 1 multicast + n unicast [+ 1 multicast] Controller1 Controller2 Controller3 DB1 DB2 DB3 DB4 DB5 DB6 DistributedVirtualDatabase@controller1 execwriterequest(abstractwriterequest) return result DistributedRequestManager Scheduler Distributed scheduling handler Write query handler Send request to all controllers DistributedVirtualDatabase@controller2 DistributedRequestManager Distributed scheduling handler Distributed load balancer handler Wait if all backends failed Scheduler Load Balancer Load Balancer Distributed load balancer handler Wait if all backends failed Wait for load balancer replies Notify controller(s) that have failed if other have succeeded DistributedVirtualDatabase@controller3 DistributedRequestManager Distributed scheduling handler Scheduler Return result Distributed load balancer handler Load Balancer Wait if all backends failed JOnAS developer workshop - c-jdbc@objectweb.org 24-23/02/2004

25 Issues with JGroups Horizontal scalability resources needed by a channel instability of throughput with UDP performance scalability TCP better than UDP but unable to disable reliability on top of TCP unable to disable garbage collection ordering implementation is sub-optimal Need for a new group communication layer optimized for cluster JOnAS developer workshop - c-jdbc@objectweb.org 25-23/02/2004

26 Horizontal scalability JGroups performance on UDP/FastEthernet JOnAS developer workshop - c-jdbc@objectweb.org 26-23/02/2004

27 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 27-23/02/2004

28 Design choices JMX for both administration and monitoring JMX server is MX4J HTTP console for basic features but not suited for monitoring Swing console for monitoring uses JMX/RMI connector Currently working on a scalable monitoring infrastructure JOnAS developer workshop - c-jdbc@objectweb.org 28-23/02/2004

29 Online Monitoring JOnAS developer workshop - c-jdbc@objectweb.org 29-23/02/2004

30 Viewing logs remotely JOnAS developer workshop - c-jdbc@objectweb.org 30-23/02/2004

31 Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop - c-jdbc@objectweb.org 31-23/02/2004

32 Current status C-JDBC 1.0 rc2 release Web site: > hits per month Downloads: >1200 per month Mailing lists: 134 subscribers Ready for production environments JOnAS developer workshop - c-jdbc@objectweb.org 32-23/02/2004

33 Ongoing work Scalable group communication middleware for clusters Needed for horizontal scalability Decouple application streams from network channels Monitoring Continuous monitoring stored in repositories Generic monitoring console for both online monitoring and monitoring history browsing Administration Fully featured administration console Graphical configuration and deployment of centralized/distributed backeds and controllers (offline/online) Dynamic reconfiguration Misc. Automated Load testing RPM packaging (JPackage) JOnAS developer workshop - c-jdbc@objectweb.org 33-23/02/2004

34 Questions & Answers Thanks to all users and contributors... JOnAS developer workshop - c-jdbc@objectweb.org 34-23/02/2004

35 Virtual Database Database Backend Connection Manager MySQL JDBC driver Java client program (Servlet, EJB,...) C-JDBC driver Virtual database Authentication Manager Recovery Log Request Manager Request Cache Database Backend Connection Manager Scheduler Load balancer MySQL JDBC driver Database Backend Connection Manager MySQL JDBC driver gives the view of a single database establishes the mapping between the database name used by the application and the backend specific settings backends can be added and removed dynamically configured using an XML configuration file MySQL MySQL MySQL JOnAS developer workshop - c-jdbc@objectweb.org 35-23/02/2004

36 TPC-W Performance (Amazon.com) 1600 Throughput in requests per minute Single DB RAIDb-0 RAIDb-1 RAIDb-2 Num ber of nodes JOnAS developer workshop - c-jdbc@objectweb.org 36-23/02/2004

37 Outline Overview Fault tolerance Performance New features Roadmap JOnAS developer workshop - c-jdbc@objectweb.org 37-23/02/2004

38 Building the initial state Octopus is an ETL tool Use Octopus to store a dump of the initial database state EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM disabled dump for initial checkpoint Octopus C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop - c-jdbc@objectweb.org 38-23/02/2004

39 Journaling Backend is All database updates are logged (SQL statement, user, transaction, ) dump for initial checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org 39-23/02/2004

40 Adding backend on the fly Add new backends while system online Restore dump corresponding to initial checkpoint with Octopus JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint disabled disabled JOnAS developer workshop - c-jdbc@objectweb.org 40-23/02/2004

41 Synchronizing backends Replay updates from the log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint disabled disabled JOnAS developer workshop - c-jdbc@objectweb.org 41-23/02/2004

42 Expanded Cluster Enable backends when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint JOnAS developer workshop - c-jdbc@objectweb.org 42-23/02/2004

43 Handling a backend failure A node fails! Automatically disabled but should be fixed or changed by administrator dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop - c-jdbc@objectweb.org 43-23/02/2004

44 Restoring a backend Restore latest dump with Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop - c-jdbc@objectweb.org 44-23/02/2004

45 Re-synchronization Replay missing updates from log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log disabled JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org 45-23/02/2004

46 Healed Cluster Re-enable backend when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org 46-23/02/2004

47 Prototype C-JDBC Management Framework Shared design JOnAS developer workshop - c-jdbc@objectweb.org 47-23/02/2004

48 Making new checkpoints Disable one backend to have a coherent snapshot Mark the new checkpoint entry in the log Use Octopus to store the dump dump for initial checkpoint... dump for dump last checkpoint for last checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop - c-jdbc@objectweb.org 48-23/02/2004

49 Making new checkpoints Replay missing updates from log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log disabled JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org 49-23/02/2004

50 Making new checkpoints Re-enable backend when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop - c-jdbc@objectweb.org 50-23/02/2004