status Emmanuel Cecchet c-jdbc@objectweb.org JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 1-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 2-23/02/2004
Motivations J2EE performance scalability bounded by database performance Database tier must be scalable fault tolerant (high availability + failover) without modifying the client application using open source databases on commodity hardware JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 3-23/02/2004
How do we want to use? end-to-end open source solution for large scale J2EE clustered application servers cost-effective solution for both small and large configurations Apache Internet MySQL JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 4-23/02/2004
Redundant Array of Inexepensive Databases RAIDb controller gives the view of a single database to the client balance the load on the database backends RAIDb levels RAIDb-0: full partitioning RAIDb-1: full mirroring RAIDb-2: partial replication composition possible JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 5-23/02/2004
C-JDBC Middleware implementing RAIDb Two components generic JDBC 2.0 driver (C-JDBC driver) C-JDBC Controller C-JDBC Controller provides performance scalability high availability failover caching, logging, monitoring, Supports heterogeneous databases JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 6-23/02/2004
JDBC Without C-JDBC Using JOnAS with C- JDBC Driver Database JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 7-23/02/2004
Using JOnAS with C- JDBC Update the datasource to use the C-JDBC driver JOnAS Driver Virtual Database Controller JDBC Driver Cache Pooling Backend JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 8-23/02/2004
What C-JDBC offers No application modification No JOnAS modification No database modification Virtual Virtual Database Database Controller Load balancing Driver JDBC Driver Cache Pooling Backend Backend recovery JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 9-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 10-23/02/2004
Inside the Controller Java client Sockets RMI program (Servlet, EJB,...) C-JDBC driver Sockets RMI Administration console XML configuration file JMX C-JDBC Controller Configuration & administration XML engine Virtual database Authentication Manager Request Manager Virtual database Authentication Manager Request Manager Recovery Log Scheduler Request Cache Load balancer Recovery Log Scheduler Request Cache Load balancer Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager Database Backend Connection Manager MySQL JDBC driver MySQL JDBC driver MySQL JDBC driver Oracle JDBC driver Oracle JDBC driver MySQL MySQL MySQL Oracle Oracle JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 11-23/02/2004
Fault tolerant recovery log Client application (Servlet, EJB,...) C-JDBC driver C-JDBC Controller Checkpointing service Recovery virtual database Application virtual database Authentication Manager Request Manager Authentication Manager Request Manager Scheduler Load balancer Recovery Log C-JDBC driver Scheduler Request Cache Load balancer Database Backend Database Backend Database Backend Database Backend Database Backend MySQL MySQL MySQL MySQL JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 12-23/02/2004
Cascading controller Client program C-JDBC driver JVM Client program C-JDBC driver JVM Client program C-JDBC driver JVM C-JDBC controller Full replication C-JDBC controller Full replication DB native JDBC driver C-JDBC driver C-JDBC driver DB 1 DB 2 C-JDBC controller Full replication DB native JDBC driver C-JDBC driver DB 3 DB 4 C-JDBC controller Full replication DB native JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org DB 5 DB 6 DB 7 13-23/02/2004
Heterogeneity support application already written for a specific [commercial] database user defined rules for on-the-fly query rewriting to execute on heterogeneous backends Java client program C-JDBC driver JVM Oracle JDBC driver Servlet container Tomcat, Jetty,... C-JDBC driver JVM C-JDBC Controller RAIDb-2 EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM MySQL JDBC driver Oracle MySQL MySQL MySQL JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 14-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 15-23/02/2004
Query caching status Cache contains a list of SQL->ResultSet Policy defined by querypattern->policy 3 policies EagerCaching: variable granularities for invalidations RelaxedCaching: invalidations based on timeout NoCaching: never cached RUBiS bidding mix with 450 clients No cache Coherent cache Relaxed cache Throughput (rq/min) 3892 4184 4215 Avg response time 801 ms 284 ms 134 ms Database CPU load 100% 85% 20% C-JDBC CPU load - 15% 7% JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 16-23/02/2004
Throughput: 3900 pages/min RUBiS - Tomcat without C-JDBC caching 1 Database 100% cpu Tomcat ~50% cpu JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 17-23/02/2004
Throughput: 4200 pages/min RUBiS - Tomcat with C-JDBC relaxed caching Tomcat ~55% cpu 1 Database ~20% cpu C-JDBC <10% cpu JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 18-23/02/2004
JOnAS and C-JDBC query RUBiS caching Throughput (req/min) No cache Response time (ms) No cache Servlet 3890 4140 744 310 JOnAS SF CMP 1.1 2947 3418 2929 2045 JOnAS CMP 2.0 2178 2450 6372 4630 JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 19-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 20-23/02/2004
Horizontal scalability JGroups for controller synchronization Groups messages for writes only Client 1 Client... Client n Client 1 Client... Client n Controller1 Controller1 Controller2 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 21-23/02/2004
Horizontal scalability Centralized write approach issues commit commit Controller1 Controller2 Controller1 Controller2 Controller2 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 DB1 DB2 DB3 DB4 commit commit???? Issues with transactions assigned to connections T1(Read) T1(Write) Controller1 Controller2 Controller1 Controller2 Controller1 Controller2 DB1 DB2 DB1 DB2 DB1 DB2 T1(R) T1(W) T1(R) T2(W) JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 22-23/02/2004
Horizontal scalability General case for a write query 3 multicast + 2n unicast DistributedVirtualDatabase@controller1 execwriterequest(abstractwriterequest) return result DistributedRequestManager Scheduler Distributed scheduling handler Write query handler Send request to all controllers Wait for schedulers replies DistributedVirtualDatabase@controller2 DistributedRequestManager Distributed scheduling handler Distributed load balancer handler Notify scheduler of completion Scheduler Load Balancer Load Balancer Distributed load balancer handler Notify scheduler of completion Send execute request order to all controllers Wait for load balancer replies Notify controllers of completion success or not DistributedVirtualDatabase@controller3 DistributedRequestManager Distributed scheduling handler Scheduler Return result Distributed load balancer handler Load Balancer Notify scheduler of completion JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 23-23/02/2004
Horizontal scalability Solution: No backend sharing 1 multicast + n unicast [+ 1 multicast] Controller1 Controller2 Controller3 DB1 DB2 DB3 DB4 DB5 DB6 DistributedVirtualDatabase@controller1 execwriterequest(abstractwriterequest) return result DistributedRequestManager Scheduler Distributed scheduling handler Write query handler Send request to all controllers DistributedVirtualDatabase@controller2 DistributedRequestManager Distributed scheduling handler Distributed load balancer handler Wait if all backends failed Scheduler Load Balancer Load Balancer Distributed load balancer handler Wait if all backends failed Wait for load balancer replies Notify controller(s) that have failed if other have succeeded DistributedVirtualDatabase@controller3 DistributedRequestManager Distributed scheduling handler Scheduler Return result Distributed load balancer handler Load Balancer Wait if all backends failed JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 24-23/02/2004
Issues with JGroups Horizontal scalability resources needed by a channel instability of throughput with UDP performance scalability TCP better than UDP but unable to disable reliability on top of TCP unable to disable garbage collection ordering implementation is sub-optimal Need for a new group communication layer optimized for cluster JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 25-23/02/2004
Horizontal scalability JGroups performance on UDP/FastEthernet JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 26-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 27-23/02/2004
Design choices JMX for both administration and monitoring JMX server is MX4J HTTP console for basic features but not suited for monitoring Swing console for monitoring uses JMX/RMI connector Currently working on a scalable monitoring infrastructure JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 28-23/02/2004
Online Monitoring JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 29-23/02/2004
Viewing logs remotely JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 30-23/02/2004
Outline Overview Advanced concepts Query caching Horizontal scalability Monitoring Perspectives JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 31-23/02/2004
Current status C-JDBC 1.0 rc2 release Web site: >100.000 hits per month Downloads: >1200 per month Mailing lists: 134 subscribers Ready for production environments JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 32-23/02/2004
Ongoing work Scalable group communication middleware for clusters Needed for horizontal scalability Decouple application streams from network channels Monitoring Continuous monitoring stored in repositories Generic monitoring console for both online monitoring and monitoring history browsing Administration Fully featured administration console Graphical configuration and deployment of centralized/distributed backeds and controllers (offline/online) Dynamic reconfiguration Misc. Automated Load testing RPM packaging (JPackage) JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 33-23/02/2004
Questions & Answers Thanks to all users and contributors... http://c-jdbc.objectweb.org JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 34-23/02/2004
Virtual Database Database Backend Connection Manager MySQL JDBC driver Java client program (Servlet, EJB,...) C-JDBC driver Virtual database Authentication Manager Recovery Log Request Manager Request Cache Database Backend Connection Manager Scheduler Load balancer MySQL JDBC driver Database Backend Connection Manager MySQL JDBC driver gives the view of a single database establishes the mapping between the database name used by the application and the backend specific settings backends can be added and removed dynamically configured using an XML configuration file MySQL MySQL MySQL JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 35-23/02/2004
TPC-W Performance (Amazon.com) 1600 Throughput in requests per minute 1400 1200 1000 800 600 400 200 0 0 1 2 3 4 5 6 Single DB RAIDb-0 RAIDb-1 RAIDb-2 Num ber of nodes JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 36-23/02/2004
Outline Overview Fault tolerance Performance New features Roadmap JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 37-23/02/2004
Building the initial state Octopus is an ETL tool Use Octopus to store a dump of the initial database state EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM disabled dump for initial checkpoint Octopus C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 38-23/02/2004
Journaling Backend is All database updates are logged (SQL statement, user, transaction, ) dump for initial checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 39-23/02/2004
Adding backend on the fly Add new backends while system online Restore dump corresponding to initial checkpoint with Octopus JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint disabled disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 40-23/02/2004
Synchronizing backends Replay updates from the log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint disabled disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 41-23/02/2004
Expanded Cluster Enable backends when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver dump for initial checkpoint JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 42-23/02/2004
Handling a backend failure A node fails! Automatically disabled but should be fixed or changed by administrator dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 43-23/02/2004
Restoring a backend Restore latest dump with Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 44-23/02/2004
Re-synchronization Replay missing updates from log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log disabled JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 45-23/02/2004
Healed Cluster Re-enable backend when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 46-23/02/2004
Prototype C-JDBC Management Framework Shared design JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 47-23/02/2004
Making new checkpoints Disable one backend to have a coherent snapshot Mark the new checkpoint entry in the log Use Octopus to store the dump dump for initial checkpoint... dump for dump last checkpoint for last checkpoint JDBC Recovery Log Octopus EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM C-JDBC Controller Recovery Log JDBC driver disabled JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 48-23/02/2004
Making new checkpoints Replay missing updates from log EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log disabled JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 49-23/02/2004
Making new checkpoints Re-enable backend when done EJB Container JOnAS, WebLogic, JBoss, WebSphere,... C-JDBC driver JVM dump for initial checkpoint... dump for dump last checkpoint for for last checkpoint JDBC Recovery Log Octopus C-JDBC Controller Recovery Log JDBC driver JOnAS developer workshop http://www.objectweb.org - c-jdbc@objectweb.org 50-23/02/2004