Persistence Performance Tips

Persistence Performance Tips Dan Bunker

Training Overview Persistence Performance Overview Database Performance Tips JPA Performance Tips Spring JDBC Performance Tips Other Tips

Prerequisites Java 6+, IDE and DB client Class Provided Pet Store App and DB Seed Previous Experience with Spring JDBC and JPA Part 1, 2 and 3 Spring Training Introduction to JPA Training Database Development 1 & 2 Previous Oracle Experience and RDBM s Understanding of Maven

Performance Problems Poor data and data structure design ORM s Reporting Poorly configured connections and database Latency and bottlenecking

Trade Off Considerations

Trade Off Considerations Reads

Trade Off Considerations Reads Paging Memory Footprint

Trade Off Considerations Reads Paging Memory Footprint Data Consolidation Normalization

Trade Off Considerations Reads Paging Memory Footprint Data Consolidation Normalization Models 2D-Arrays

(cont.)

(cont.) Writes

(cont.) Writes Graph Writes SQL Cascade Complexity

(cont.) Writes Graph Writes SQL Cascade Complexity ORM Batching Footprint and Performance

(cont.) Writes Graph Writes SQL Cascade Complexity ORM Batching Footprint and Performance Validation FK s and Constraints

Thought Scenario

Thought Scenario Urban Pop. ~ 3.5 Million Pers./Sq Mile ~ 7250

Thought Scenario Urban Pop. ~ 3.5 Million Pers./Sq Mile ~ 7250 Urban Pop. ~ 8.4 Million Pers./Sq Mile ~ 27550

Thought Scenario Urban Pop. ~ 3.5 Million Urban Pop. ~ 8.4 Million Pers./Sq Mile ~ 7250 Pers./Sq Mile ~ 27550 Which city is more performant? Which city has the best throughput?

Improving Database Performance In Depth

Planning to Plan Explain Plan shows table scans and index usage shows join plans and performance cost Oracle uses a shared temporary table for plans Output only shows when in same session as explain

Planning to Plan Explain Plan shows table scans and index usage shows join plans and performance cost Oracle uses a shared temporary table for plans Output only shows when in same session as explain explain plan into sys.plan_table$ for select id, name from animal; select * from table(dbms_xlpan.display( sys.plan_table$ ));

Explain Output

Compound Join Output

Optimizer Statistics Database details help the optimizer choose the best explain plan by looking at: Table Statistics (# rows, #blocks, ave. row length) Column Statistics (distinct values, nulls, distribution) Index Statistics (# of leafs, levels, clustering) System Statistics (I/O and CPU performance)

Performance Improvement #1 Create and Maintain Appropriate Indexes

Indexes

Indexes Data Structure

Indexes Data Structure Copy or Subset of the Table

Indexes Data Structure Copy or Subset of the Table Improves Data Retrieval

Indexes Data Structure Slows down writes Copy or Subset of the Table Improves Data Retrieval

Indexes Data Structure Slows down writes Copy or Subset of the Table Consumes more storage space Improves Data Retrieval

Indexes Data Structure Slows down writes Copy or Subset of the Table Consumes more storage space Improves Data Retrieval Easy to add

Which Columns to Index?

Which Columns to Index? Columns frequently used in where clauses

Which Columns to Index? Columns frequently used in where clauses Columns used for table joins (i.e. FK s)

Which Columns to Index? Columns frequently used in where clauses Columns used for table joins (i.e. FK s) Add indexes judiciously on heavily modified cols.

Which Columns to Index? Columns frequently used in where clauses Columns used for table joins (i.e. FK s) Add indexes judiciously on heavily modified cols. Columns with high selectivity (i.e. High percentage of rows with the same value)

Checking Indexes

Checking Indexes select c.index_name, c.table_name, c.column_name, i.uniqueness, i.index_type from SYS.ALL_INDEXES i join SYS.ALL_IND_COLUMNS c on i.index_name = c.index_name where i.table_name = 'ANIMAL'

Checking Indexes select c.index_name, c.table_name, c.column_name, i.uniqueness, i.index_type from SYS.ALL_INDEXES i join SYS.ALL_IND_COLUMNS c on i.index_name = c.index_name where i.table_name = 'ANIMAL' INDEX_NAME TABLE_NAME COLUMN_NAME UNIQUENESS INDEX_TYPE ----------- ---------- ----------- ---------- ---------- SYS_C005166 ANIMAL ID NONUNIQUE NORMAL Elapsed Time: 0 hr, 0 min, 0 sec, 2 ms.

Lab 1 Analyze Query and Recommend Index

Assignment Given the following SQL Statement: select * from Animal a join Animal_Country ac on a.id = ac.animal_id join Country c on ac.country_id = c.id where c.name like 'Un%'; Run explain plan, analyze output and determine if adding a column index to any tables may be beneficial

Assignment Notes select * from table(dbms_xplan.display('sys.plan_table$')); Query to display current indexes for a given table Query to display explain plan output select c.index_name, c.table_name, c.column_name, i.uniqueness, i.index_type from SYS.ALL_INDEXES i join SYS.ALL_IND_COLUMNS c on i.index_name = c.index_name where i.table_name = 'ANIMAL'

Solution 1 Analyze Query and Recommend Index

Adding Indexes Add indexes as needed with a db migration

Adding Indexes Add indexes as needed with a db migration CREATE INDEX ac_animal_id_fk_ind ON animal_country(animal_id) TABLESPACE users STORAGE (INITIAL 20K NEXT 20k PCTINCREASE 75);

Improving JPA Performance In Depth

Common JPA Issues Eager Loading vs. Lazy Loading Displaying data from 2 or more table relations in 1 view N+1 Queries Persistence Context lifecycle

Pet Store Admin View

Pet Store Admin View

What Really Happens

What Really Happens

How to solve this Create a PetDto constructor that doesn t initialize the classifications Create a Projection Query with a DTO Utilize JPA fetches Other ways we ll get to in a bit

Performance Improvement #2 Use JPA Fetches

Join Fetch Loads the primary entity and an associated entity in 1 query Entity models get hydrated appropriately JPAQL only First place to look when solving N+1 problem

Join Fetch Syntax Add fetch keyword to JPAQL joins If the association is not present in the JPAQL statement you can add it simply for the fetch

Join Fetch Syntax Add fetch keyword to JPAQL joins If the association is not present in the JPAQL statement you can add it simply for the fetch SELECT o FROM Order o WHERE (o.status) =?1 ORDER BY o.id

Join Fetch Syntax Add fetch keyword to JPAQL joins If the association is not present in the JPAQL statement you can add it simply for the fetch SELECT o FROM Order o WHERE (o.status) =?1 ORDER BY o.id SELECT o FROM Order o JOIN FETCH o.orderdetails d WHERE (o.status) =?1 ORDER BY o.id

Lab 2 Fix N+1 Query Problem

Assignment Fix the Pet Store Admin search by removing the classification N+1 queries Hint: A blank search calls the AnimalRepository.findAnimalsWithName(String name) method

Assignment Fix the Pet Store Admin search by removing the classification N+1 queries Hint: A blank search calls the AnimalRepository.findAnimalsWithName(String name) method SELECT a FROM Animal a WHERE UPPER(a.name) LIKE upper(?1) ORDER BY a.name

Solution #2 Fix N+1 Query Problem

Fetch Notes Setting the FetchType on an annotation won t fix N+1 issues JOIN FETCH becomes problematic with Hibernate provider when fetching multiple associations Weigh the performance cost of the N+1 queries vs. the cost of the join query

Performance Improvement #3 Projection Queries

Data Packet Scenarios View Tier Persistence Tier

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User JpaRepository User

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User User User User JpaRepository User

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User User User User JpaRepository User UserDTO

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User User User UserDTO UserDTO UserDTO User JpaRepository UserDTO User

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User User User UserDTO UserDTO UserDTO User JpaRepository UserDTO User UserDTO UserDTO UserDTO

Data Packet Scenarios View Tier Persistence Tier @Repository @Entity User User User UserDTO UserDTO UserDTO User JpaRepository UserDTO User User JSON UserDTO UserDTO UserDTO

Projection Queries

Projection Queries Custom Select values to hydrate a DTO

Projection Queries Custom Select values to hydrate a DTO Natively supported by JPAQL

Projection Queries Custom Select values to hydrate a DTO Natively supported by JPAQL Criteria Queries support but more verbose

Projection Queries Custom Select values to hydrate a DTO Doesn t always solve N+1 problem Natively supported by JPAQL Criteria Queries support but more verbose

Projection Queries Custom Select values to hydrate a DTO Doesn t always solve N+1 problem Natively supported by JPAQL No compile time checking creating possible runtime issues when refactoring Criteria Queries support but more verbose

Projection Queries Custom Select values to hydrate a DTO Doesn t always solve N+1 problem Natively supported by JPAQL No compile time checking creating possible runtime issues when refactoring Criteria Queries support but more verbose Severs persistence context association preventing unwanted graph follows

Projection Syntax Create DTO constructor to match your needs Instantiate DTO in select clause

Projection Syntax Create DTO constructor to match your needs Instantiate DTO in select clause @Query("SELECT new org.lds.stack.petstore.admin.model.petdto(a.id, a.name, a.description, a.price) FROM Animal a WHERE UPPER(a.name) LIKE upper(?1) ORDER BY a.name")

Lab 3 Add Pet Search Using Projection Queries

Assignment Add a PetDto constructor to take id, name, description, and price Add a Projection Query to AnimalRepsitory Change PetManagerFacadeImpl.searchForPetsBy to use the new projection query

Assignment Add a PetDto constructor to take id, name, description, and price Add a Projection Query to AnimalRepsitory Change PetManagerFacadeImpl.searchForPetsBy to use the new projection query SELECT new org.lds.stack.petstore.admin.model.petdto(a.id, a.name, a.description, a.price) FROM...

Solution #3 Add Pet Search Using Projection Queries

Projection Query Notes Can get messy when passing in many constructor parameters DTO is another class that needs to be created and maintained besides the entity model Similar strategy to Spring JDBC but utilizes JPAQL instead of SQL

Improving Spring JDBC Performance In Depth

Spring JDBC Issues Poorly written SQL statements DTO s and domain holder objects are too flat Relationship management is verbose

Spring JDBC Issues Poorly written SQL statements DTO s and domain holder objects are too flat Relationship management is verbose Other issues that the stack has addressed Connection Pooling Transactions and Statement Caching

JDBC Performance Most problems will be in your SQL

JDBC Performance Most problems will be in your SQL Batch deletes and updates

JDBC Performance Most problems will be in your SQL Batch deletes and updates Manage cache of commonly read data

JDBC Performance Most problems will be in your SQL Tweak the fetchsize when dealing with large amounts of data Batch deletes and updates Manage cache of commonly read data

Performance Improvement #4 Set Fetch Size

JDBC Fetch Size Determines how many results to pull at a time in the ResultSet (cursor)

JDBC Fetch Size Determines how many results to pull at a time in the ResultSet (cursor) JdbcTemplate defaults to 0

JDBC Fetch Size Determines how many results to pull at a time in the ResultSet (cursor) JdbcTemplate defaults to 0 Trade off of performance vs. memory

JDBC Fetch Size Determines how many results to pull at a time in the ResultSet (cursor) Oracle Driver has generally defaulted to 10 but can vary JdbcTemplate defaults to 0 Trade off of performance vs. memory

JDBC Fetch Size Determines how many results to pull at a time in the ResultSet (cursor) Oracle Driver has generally defaulted to 10 but can vary JdbcTemplate defaults to 0 Trade off of performance vs. memory Poorly set Fetch Sizes can make queries last 2 to 4 times longer

Code Snippet

Code Snippet public List<Example> findbyname(string name) { } jdbctemplate.setfetchsize(100); List<Example> examples = jdbctemplate.query("select * " + " from EXAMPLE where EXAMPLE_NAME like?", new ExampleRowMapper(), name); jdbctemplate.setfetchsize(0); return examples;

Other Performance Solutions In Depth

Last Resorts

Last Resorts Caching

Last Resorts Caching Views

Last Resorts Caching Views De-normalization

Last Resorts Caching JPA: Native SQL Views De-normalization

Last Resorts Caching JPA: Native SQL Views JPA: Manually manage associations De-normalization

Last Resorts Caching JPA: Native SQL Views JPA: Manually manage associations De-normalization Stored Procedures

Caching

Caching Utilize stack cache module Don t use 2nd level caching (JPA) Don t use for highly volatile data Think about memory footprint

Caching Utilize stack cache module Don t use 2nd level caching (JPA) Don t use for highly volatile data Think about memory footprint For large amounts of data caching or large data sets look at using a caching server

Views

Views Table A Table B Table C

Views Table A Table B View ABC Table C

Views Table A Table B View ABC Table C @Repository

View Pitfalls

View Pitfalls Altering your persistence architecture Some limitations with SQL on views View caches vs updates (materialized views versus standard views) De-normalization can lead to duplicate record sets in the view typically forcing a where clause on all queries using that view

Note on SQL select /*+ FIRST_ROWS(10) */ * from Animal a join Animal_Country ac on a.id = ac.animal_id join Country c on ac.country_id = c.id where c.name like 'Un%';

Note on SQL Utilize the explain plan to see how joins are being executed Provide SQL Hints to help the optimizer select /*+ FIRST_ROWS(10) */ * from Animal a join Animal_Country ac on a.id = ac.animal_id join Country c on ac.country_id = c.id where c.name like 'Un%';