The NoPlsql and Thick Database Paradigms Part 2: Adopting ThickDB Toon Koppelaars Real-World Performance Oracle Server Technologies Bryn Llewellyn Distinguished Product Manager Oracle Server Technologies
Who Are We? Part of Oracle ecosystem since 1987 Developer, DBA, Architect, Problem Solver Currently part of Real-World Performance group Author of TheHelsinkiDeclaration.blogspot.com All about ThickDB @ToonKoppelaars Distinguished Product Manager: PL/SQL and EBR Author of "Why use PL/SQL?" blogs.oracle.com/plsql-and-ebr/entry/why_use_pl_sql @BrynLite 2
Database Applications (from Part 1) User Interfaces SQL SQL SQL Business Logic Software Interfaces SQL SQL Table Data Store SQL Conceptually 3 tiers Exposed functionality via services GUI's for human interaction REST, or otherwise, for software interaction Business logic Procedural code composing Q's + TX's Data store, relational database 4
Persistence Layer or Processing Engine? SQL SQL SQL SQL SQL SQL DBMS DBMS SQL SQL SQL SQL No PL/SQL Single table, row-by-row SQL NoPlsql PL/SQL All kinds of SQL ThickDB 5
Focus of This Part 2 You want to adopt ThickDB approach for development of new application Now, how to do that successfully? Critical success factors: Technical Understand various concepts, architecture etc. How to prevent one big pile of PL/SQL code Organizational Get project roles right 6
Roadmap 1 2 3 4 Organizational - Project roles Technical - Architecture of web applications Technical - Code classification and architecture inside database Concluding thoughts 7
Organizational: Project Roles "Full-stack" developers with solid Database-design, SQL, PL/SQL, *and* Current UI technology skills, are hard to find High technology change rate You'll inevitably have separate roles in your project Database developers easier to hire without UI skills Hip JavaScript UI developers easier to hire without DB skills 8
Organizational: Focus For Each Role Focus for each: Database developers strong (set-based) SQL and PL/SQL skills Focus on maintainability, performance and scalability UI developers strong html/css/javascript skills Focus on extremely friendly UI experience (UX) Separate focus allows both roles to be performed at highest level Database developer skills are more rare, but one database developer can support multiple UI developers 9
Organizational: Two Worlds Must Cooperate Separate focus, but must closely cooperate Requires good communication skills Database developers will need to understand UI-developers' world Not vice-versa: they will be "decoupled" via API's Know enough architectural concepts to communicate effectively with them, to jointly design API-layer Database developers, must be in the lead here Why? they own database design and business logic Need to detect API requests that would leave BL in UI 10
The New Emerging World Creates Opportunity V+C physically de-coupled now from M 11
Roadmap 1 2 3 4 Organizational - Project roles Technical - Architecture of web applications Technical - Code classification and architecture inside database Concluding thoughts 12
Architecture of Web Applications In web applications, users: Tick boxes Select values from dropdown LoV's Enter/change editable data items Click on hyperlinks Press buttons Do something that triggers (browser-side) JavaScript These can execute Ajax calls Popular in "Single Page Applications" (SPA's) Partial page (re)rendering These typically initiate request from browser to server Asynchronous Javascript And XML Understanding Technical Architecture 13
Browser Server Requests Result in Page Transitions Click on link Press button Initiate Ajax call Browser Browser Browser http-get New page html http-post Form submit New page html Ajax request XML for re-render Web server Web server Web server Understanding Technical Architecture 14
Point to Be Made Understand where, when and why UI developer will submit requests For each request that browser can initiate you will need to agree upon some API that handles it Web server A P I Web server acts as proxy: it just translates request to DB-API call Understanding Technical Architecture 15
Page Flows These http-get, http-post and Ajax requests result in page transitions Which in turn lead to application "page flows" 16
Page Flow Network Customers Edit Customer Login Start Page Orders Edit Order Understanding Technical Architecture 17
Incoming Flow Determines What Should Be Displayed Show next/previous ten orders Show 1 st ten orders of this customer E.g. default show 1 st ten "open" orders Edit Customer Start Page Orders Edit Order Refresh list that was shown So here we have four "render" use-cases for Orders-page For each case DB-API is tasked to produce list of orders to be displayed Understanding Technical Architecture 18
Transactions Typically Attached to Outgoing Flows Delete-button pressed Update-button pressed Orders Edit Order Two process use-cases for Edit-Order page For each case DB-API is tasked to compose and execute transaction Understanding Technical Architecture 19
Render and Process Use-Cases per Page You'll need to discuss per page What are the render use-cases? What are the process use-cases? And per use-case agree upon signature: What are API inputs and outputs Understanding Technical Architecture 20
The Connection Pool Application threads use "connection pool" to access database Some number of pre-instantiated Java objects Each owning database session Understanding Technical Architecture 21
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Servlet execution threads working on browser requests Connection pool C1 C2 C3 Pre-spawned connection pool with three idle connections JDBC JVM S1 S2 S3 Idle database sessions All under same username Understanding Technical Architecture 22
Connection Pool Thread 6 requires database access T1 T2 T3 T4 T5 T6 T7 T8 Connection pool C1 C2 C3 JDBC JVM S1 S2 S3 Understanding Technical Architecture 23
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Thread 6 requests and acquires free connection from pool C1 C2 C3 JDBC Connection pool JVM C2 now unavailable for other threads S1 S2 S3 Understanding Technical Architecture 24
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Connection pool C1 C2 C3 JDBC JVM Thread 6 submits SQL via C2 S1 S2 S3 DB session S2 wakes up from idle state Understanding Technical Architecture 25
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Connection pool C1 C2 C3 JDBC JVM S1 S2 S3 S2 executes SQL statement Understanding Technical Architecture 26
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Connection pool C1 C2 C3 JDBC JVM S2 sends result back to C2 S1 S2 S3 S2 idle again Understanding Technical Architecture 27
Connection Pool T1 T2 T3 T4 T5 T6 T7 T8 Connection pool C2 sends result back to T6 C1 C2 C3 JDBC JVM S1 S2 S3 Understanding Technical Architecture 28
Connection Pool T6 releases C2 back to pool T1 T2 T3 T4 T5 T6 T7 T8 C1 C2 C3 JDBC Connection pool JVM C2 available again for other threads S1 S2 S3 Understanding Technical Architecture 29
Connection Pool These connection/session pairs are sequentially used by requests from different browsers Understanding Technical Architecture 30
Points to Be Made Threads should follow strict "request dosql release" pattern Each database request must either: Constitute of queries only, or Represent completed transaction (committed or rolled back) No enduser specific residue must be left in released connection's database session Temp table contents V$context contents Values in global package variables NLS settings Unclosed cursors Unhandled exceptions, during "dosql", should not break release step Leading to connection leakage 31
Roadmap 1 2 3 4 Organizational - project roles Technical - Architecture of web applications Technical - Code classification and architecture inside database Concluding thoughts 32
First Things First: The Foundation "Proper" = two-fold 1. It must fit business requirements 2. It must follow sound relational practices Proper relational database design 1) is up to you talking to business owners and understanding their problem 2) is about education 33
Recommended Practices For a Sound Database Design Use 3 rd normal form Avoid structural redundancies Don't design incidental redundancies E.g. Total value of orderlines at order level Surrogate key 2 nd business key Put different things in different tables Avoid different meaning of column value depending on row Put special things in separate tables (subtypes) Avoid lists of null-able columns due to non-applicability Never allow nulls in columns involved in data integrity constraints Clarification Choose your table- and column-names wisely They are the first steppingstone to understanding your DB design 34
Data Integrity Constraints The relational model's meaning: Constraints are second stepping stone to understanding your database design The broader generic concept used in E.F Codd s relational model of data Everything that constrains the data that is allowed I.e. much more that just: CHECK, PRIMARY KEY, UNIQUE KEY, FOREIGN KEY SQL standard includes SQL Assertions for these This is all about predicate logic Boolean expressions that must be true at all times See the book 35
SQL Assertions Example Last DB design recommended practice: all columns referenced should be NOT NULL create assertion at_most_one_president as check ((select count(*) as num_president from EMP e where e.job = 'PRESIDENT') <= 1 ) Not supported yet, but Excellent way to document constraints! 36
https://community.oracle.com/ideas/13028
Point to Be Made Your database design will have many SQL assertions You will write lot of code to enforce them Enforcement is required during transactional API's All code written to enforce SQL assertions: Data Logic code (DL) Different approaches Table triggers Table API's (TAPI's) Rule packages 38
Code Classification Use "du-jour" technology User Interfaces Software Interfaces Procedural code with embedded queries Business TX Logic Business Query Logic Business SQL SQL Logic Data SQL SQL Logic SQL SQL Table Data Store Procedural code with embedded DML and error handling Treat separately from TX-BL Challenge here is: DRY Specify all declarable constraints 39
Code Classification API Layer User Interfaces API Layer Software Interfaces Business TX Logic Business Query Logic Business SQL SQL Logic Data SQL SQL Logic SQL SQL Table Data Store 40
Code Classification Apply Common Sense DRY User Interfaces API Layer Software Interfaces Business TX Logic Business Query Logic Business SQL SQL Logic Data SQL SQL Logic SQL SQL Shared Views / Functions Table Data Store Business specific - Data views - Data derivations - Computations 41
Code Classification Key Message These will then become These will then become straightforward straightforward procedural code User Interfaces API Layer Software Interfaces Business TX Logic Business Query Logic Business SQL SQL Logic Data SQL SQL Logic SQL SQL Shared Views / Functions Table Data Store Make sure that these two are completely separated from other business logic code 42
End of Nineties Wisdom Q: How do I prevent one big pile of PL/SQL code? Adversaries argument A: Materialize code classification such that classes, and modules within them, are separately, Visible Maintainable Versionable Etc. 43
Schema Architecture: Horizontal Materialization Introduce multiple database schemas Four horizontal layers 1) "Connect schema": object-less schema, used by connection pool to connect to DB 2) "API schema" : holds all API objects jackets on top of code schema objects 3) "Code schema" : holds all business logic and data logic 4) "Storage schema" : holds all tables and declarative constraints 44
Schema Architecture T1 T2 T3 T4 T5 T6 Connection pool JVM C1 C2 C3 JDBC Connect schema API schema Code schema Storage schema S1 S2 S3 API objects Business logic Tables 45
Remember the Page Network Incoming flows that render Page X Context B Page X Outgoing flows that initiate TX 46
Vertically Materialize API's Per Page Page X Create package per page Use page-name/id in its name Render API's Submit API's Add procedure per incoming / outgoing flow U S E C A S E 1 U S E C A S E 2 U S E C A S E 3 U C 1 U C 2 Issue in Page X submit, usecase 2? Impact analysis (dependencies) 47
API Object Signature All API's will be packaged procedures Accept and return repeating data via parameters of type: XML JSON (nested) IBI's of varchar2's Index-by-plsinteger record-arrays Accept and return scalar data via parameters of type: Varchar2 48
Roadmap 1 2 3 4 Organizational - project roles Technical - Architecture of web applications Technical - Code classification and architecture inside database Concluding thoughts 49
Summary We've seen 15 years of MVC in middle-tier With persistence layer role for database Now strong move towards V+C in browser using JavaScript frameworks/libraries M "somewhere else" Both tied together using REST This is opportunity for ThickDB approach to revive 50
Summary Separate UI (V+C) and database (M) project roles With DB-developers in the lead who understand technical architecture UI DB Decouple UI via API's that hide all BL+DL Materialize code classes Build UI use-cases vertically Apply common-sense DRY Deploy horizontally using different schemas 51
52
53