WHITE PAPER SEPTEMBER Reactive Rules for Database Logic. Agility, Re-Use and Performance

Transcription

1 WHITE PAPER SEPTEMBER 2018 Reactive Rules for Database Logic Agility, Re-Use and Performance

2 2 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Executive Summary Introduction Since the advent of relational databases, enterprises have been devising methods for extracting data in a useful fashion for application development. However, traditional data-driven app dev has been relatively unchanged since the first application of object-oriented functional programming to these systems. Even with the recent popularity of API-driven interfaces, much of the development burden has involved back-end effort. Creating application back-ends involves interacting with databases to retrieve data through getters and setters that map to SQL statements on single tables, writing more complex SQL to do manual JOINs across database tables, and defining database triggers or client-side logic to manage relationships between fields and maintain data consistency. While some progress has been made using code generation techniques to automate single-table API creation, the results are complex, hard to maintain and inadequate for real-world use cases that require application logic. At the same time, the need for agile app development in the application economy has made the ability to rapidly deliver robust data-driven APIs more important than ever. These initiatives are also being driven by lines of business rather than IT organizations, meaning the API owners are likely to be more familiar with a spreadsheet than with server-side programming languages. The way we build application back-ends, interact with data sources and define logic needs to evolve to keep up with these ever-changing requirements. This white paper takes a fresh look at defining application logic and interacting with databases using Reactive Logic rather than traditional functional code-based approaches. It explores the concept of reactive programming and how this paradigm can be applied to declarative rules for maintaining relationships across data tables. It demonstrates the power of Reactive Logic to simplify database interaction and logic definition for business users while maintaining robust performance and optimization characteristics. And it shows how CA Technologies will forever change the way you look at application creation.

3 3 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com What is Reactive Programming? Reactive programming is a declarative approach where variables defined by expressions automatically react to changes in referenced values. As well-explained in this Wikipedia entry, reactive programming is a declarative not imperative (aka procedural) approach for automatic propagation of changes: In computing, reactive programming is a programming paradigm oriented around data flows and the propagation of change. This means that it should be possible to express static or dynamic data flows with ease in the programming languages used and that the underlying execution model will automatically propagate changes through the data flow. For example, in an imperative programming setting, a: b + c would mean that a is being assigned the result of b + c in the instant the expression is evaluated. Later, the values of b and c can be changed with no effect on the value of a. In reactive programming, the value of a would be automatically updated based on the new values This is an abstract definition that can be adapted to various disciplines as described below. Regardless, the system needs to establish watches on the referenced values and react when changes are observed. The watch mechanism may be automatic or manual wherein you must explicitly invoke operations to establish watches. Current industrial use Various vendors have been active in the field of reactive programming and it remains a popular topic on developer sites such as Stack Overflow, but by far the most famous commercial application is the spreadsheet. Your cells are defined by values referencing other cells the referencing cell reacts to changes in the referenced cells by re-computing its value. This process can of course chain, enabling quite complex computational systems to be defined. Reactive Rules for Database Logic The powerful reactive programming model is perfectly suited to address the change propagation required for transaction processing, that is, when the server receives change (Insert, Update, or Delete) requests. This results in reactive rules that can express business logic within an application without writing functional code. The key approach borrows heavily from domain-driven design to provide declarative domain logic and is described below.

4 4 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com 1. Base tables/columns/roles are variables. The relational database schema is the object model used to define the variables that can be referenced in reactive programming. So, you can refer to customer.balance. This can be extended to provide for attributes not defined in the schema but that is beyond the scope of this article. Real-world complexity requires that expressions be able refer to data in other tables, so you can refer to purchaseorder.customer. balance, or purchaseorder.lineitems. These could potentially represent one-to-many relationships across tables. For example: Parent role we use the foreign key constraint name for references to the one side (as in purchaseorder.customer). Child role the table name is for the many side (as in purchaseorder.lineitems). Actual support provides customized child role names but that is beyond the scope of this document. 2. Define multi-table expressions for database columns. These can be simple expressions, like this, to compute the amount for a Lineitem row : return row.price * row.quantity. Expressions might be conditional, like: if (row.qty_ordered <= 6) return row.product_price * row.qty_ordered; else return row.product_price * row.qty_ordered * 0.8. Support is also provided for multi-table expressions (crucial for multi-table transactions), such as: customer.balance = sum(purchaseorder.amount_total where paid = false). 3. Establish automatic watches on dependent data. This requires the system parse the expressions to determine the referenced data, which is then watched. 4. Recalculate multi-table. The system must recalculate the referencing data when the watched data changes. Note that changes must be interpreted broadly: not just updates but also inserts and deletes. The real power occurs when you consider multi-table derivations. For true declarative/reactive support, you should not need to code SQL commands to access the data this should be fully automated. So, deleting a purchaseorder should adjust the balance. 5. Automate SQL. SQLs required for multi-table recalculation should be automated. 6. Optimize SQL handling. Which brings us to optimization. The agility/clarity advantages of reactive logic provide little value if there is a performance penalty. SQL statements are far more expensive than arithmetic, so for enterprise performance; the system needs to optimize update processing in much the same manner as we rely on relational database to optimize retrievals. The system must eliminate and optimize SQL with techniques such as:

5 5 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Pruning no SQL should occur if the watched data is not altered (e.g., changing an order s date should not require access to the customer). Adjustment SQLs are expensive but aggregate SQLs are even more so. Accordingly, recalculations of aggregates (e.g., sums) should be done with one row updates. Observe that aggregates can nest. So, a simple non-optimized transaction to pay an order would require a select sum of all the orders and items. Optimized adjustment logic can reduce these SQLs by orders of magnitude. This is very important since it provides the basis for real-time analytics, such as real-time accounting (instant posting of new transactions) and real-time medical records. Caching an order transaction with several line items should issue one adjustment to the order, not one per item. 7. View/resource mapping to objects. Critical view/resource mapping to objects is required so you can define views (or RESTful resources) optimized for program use and reuse the reactive logic associated with the basic table. 8. Chain with automatic ordering. The power of spreadsheets is that derived cells can be referenced in other cells, one change can naturally propagate through the data. The same is required for database logic. Of course, more complex dependencies require ordering. For example: a = b+c; b = c+d requires that, on a change to c, b be calculated before a. 9. Determine imperative escapes. A good question is, How much of my database logic can be addressed with reactive programming? The real-world answer is more than you think (in excess of 95 percent) but not 100 percent. So we need an imperative escape. This is best provided by a familiar event model, where you define table events that are called on inserts, updates and deletes. You might use events to send messages/ , start a business process, etc. The real power is that these events are typically predicated on the reactive derivations. The language used for event logic is of course critical. The consensus choice here is JavaScript the one language you ll probably need to use no matter what other choices you make. 10. Determine transactional support. Basic database integrity requires that multi-updates are atomic they are rolled back if errors occur. This applies to imperative multi-table transactions as well as those designed with reactive programming. 11. Determine validation support. Define a list of expressions for a table which must be true for a transaction to succeed (e.g., balance <= credit_limit), else an exception is raised.

6 6 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Architecture While this paper is primarily focused on concepts, reactive database logic is only interesting if it actually runs. It can do so in a number of ways: Database support Native database support could be provided through DDL extensions with corresponding runtime support. Many advantages but a long process to add to the SQL standard and it would still not provide natural support for remote data. Code generation of triggers and stored procedures This is, after all, what triggers do, so it would clearly be possible to generate these. This provides excellent active enforcement but does not deal well with integrating multiple databases. App servers Traditional app servers also enforce business logic with support for integrating multiple databases. The language-specific interfaces, however, are not architecture independent and the solution would be burdened by a significant coding overhead and low re-use. API Servers A more recent approach is to build API servers that provide REST-based interfaces around these reactive logic rules, typically with JavaScript support for accessing multiple databases and external services. Reactive logic example This sample problem illustrates the use and operation of reactive programming. Logic is expressed as a set of spreadsheet-like expressions that define what your data means. Consider the use case, place order, with the requirement that a customer s balance may not exceed their credit limit. The cocktail napkin requirements are illustrated above. The following logic, which looks rather like a requirements document, should be fully executable: Validation logic: Validations are expressions that must be satisfied to commit transactions (else an exception is thrown). Far beyond simple single-attribute validations, the requirement is multi-attribute validations, such as this Customer Validation: balance < credit_limit.

7 7 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Derivation logic: Validations are the most familiar form of business logic, but derivations and events are the most valuable for dependency management and re-use, as noted below. Derivations here include multi-table derivation rules such as in this rule which defines the balance as the sum of the unpaid order totals: balance is sum(purchaseorder.amount_total where paid = false). Derivations are conceptually similar to spreadsheet cell formulas. Note that: Derivations are multi-table, as in the example above. Derivations can chain, both to validations and other derivations: Purchaseorder.amount_total = sum(lineitem.amount) Lineitem.amount = qty_ordered * product_price Lineitem.product_price = copy(product.price) Chaining, ordering and dependencies are automatically handled (further described below). Derivations can include if/else logic, state transition logic and reference related data. Derivations are highly optimized to eliminate/optimize SQLs. For example, changing an order s date does not retrieve the customer and paying an order adjusts the customer with a one row update not an aggregate query. Declaring Logic You declare the logic in a browser-based IDE as shown below. Dialogs are provided to make it simple to define (point and click) and the system produces this documentation automatically.

8 8 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Reactive in action Dependency management means that when the client submits some altered data, the server detects all changes and adjusts any dependent data including related tables. This incurs ordering and requires optimization. The diagram below illustrates the server processing for the change in line item quantity (represented by the green circle at the bottom): The amount is dependent on the quantity, so it is recomputed. The amounttotal is dependent on the amount, so it is adjusted. Note this may require a SQL lookup (though the data might be in the cache) and SQL operations are pruned if the amount is not changed. The balance is dependent on the amounttotal, so it is adjusted. The result balance is credit checked. If an error is detected, the transaction is rolled back and an exception is returned. The same processing is automatically applied to all use cases, as described below in the reactive reuse section. Reactive agility By the time you boil off change detection and change propagation and SQL, you have eliminated all the noise. What is revealed is the pure, crystallized logic. As it turns out, this noise is most of the code of a traditional program. The five lines of reactive rules require literally 500 lines of imperative code. This proportion holds in fact improves as system size increases. You are operating at a two orders of magnitude higher level of abstraction.

9 9 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Active integrity Imperative code runs when you call it. Code for integrity may exist but it is of no value unless it s called in every required circumstance. So, verifying compliance entails checking all the paths of all the existing code. By contrast, you don t call reactive logic. You just state it. It is the system s job to react as required in all required circumstances. So, if you want to verify compliance, just scan the logic. If it s there, you can be certain it is being applied. Reactive reuse The customer.balance rule, perhaps declared for place order, simply states that its value is the sum of the unpaid orders. This design intent is encapsulated into purchaseorder and then automatically re-used over all related transactions. So, the balance is increased if an order is added, decreased when it is deleted or paid and so forth. Put another way, logic is associated with your tables, not with a specific transaction, request type or method. This is what enables logic re-use. So our logic above, perhaps conceived for place order, is automatically re-used for all these related transactions: Place order the balance is increased. Change customer credit limit. Delete order the balance is reduced. Pay order the balance is reduced. Un-pay order the balance is increased. Reassign order to a new customer new customer balance is increased, old balance is decreased (for unpaid orders). Reassign a line item to a different product. Add a line item. Delete a line item. Change line item quantity. Change line item price. This design one/solve many applies to maintenance as well; when you change your logic, the system automatically reflects it in all the relevant transactions. Reactive objects factor logic from the methods (still clumsy imperative code) into declarative reactive expressions: Maintenance Imperative programming is distinguished by ordering your code must be executed in a correct order. The pain of this is most evident in the archaeology of maintenance, where most of the time is spent studying existing code to determine where to insert a few new lines. This entire process is eliminated by automatic ordering. You just alter the logic and the system computes a new execution order (and optimization strategy) based on the new dependencies.

10 10 WHITE PAPER REACTIVE RULES FOR DATABASE LOGIC ca.com Automatic documentation As shown in the earlier screenshot, this logic is virtually a direct entry of the cocktail napkin spec. So logic is not only executable; it is transparent documentation that enables business users and IT to partner and find missing/incorrect elements. Reactive performance The optimizations discussed above pruning, caching and adjustment are applied to every maintenance change. So, unlike conventional systems that slow down over time due to maintenance patches, reactive systems sustain a consistently high level of performance. Conclusion Reactive programming concepts have been around for a long time and were first applied to the spreadsheet dependency management done right by computers automatically rather than by programmers who don t appreciate doing the repetitive coding again and again. CA Technologies was the first to apply reactive programming techniques to databases and bring simplicity of the spreadsheet to API development with specs that can almost be executed as-is, live APIs that can be understood by business and programming staff alike and an easier maintenance process and automatic documentation. In addition, JavaScript is provided to enhance and complement reactive rules to ensure 100 percent coverage of application requirements. Connect with CA Technologies CA Technologies (NASDAQ: CA) creates software that fuels transformation for companies and enables them to seize the opportunities of the application economy. Software is at the heart of every business, in every industry. From planning to development to management and security, CA is working with companies worldwide to change the way we live, transact and communicate across mobile, private and public cloud, distributed and mainframe environments. Learn more at ca.com. Copyright 2018 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. This document does not contain any warranties and is provided for informational purposes only. Any functionality descriptions may be unique to the customers depicted herein and actual product performance may vary. CS _0918