Database In-Memory Workshop

Transcription

1 Database In-Memory Workshop Student Workshop Lab Guide Creation Date [September 2014] Last Modified Date - 11/10/2014 Comments? Questions? Stay Connected! osc_us@oracle.com #osc_us OracleSolutionCenter

2 Table of Contents 1. WORKSHOP OVERVIEW WORKSHOP ENVIRONMENT SETUP MONITORING AND QUERYING THE IN-MEMORY COLUMN STORE LAB SETUP AND MONITORING THE IN-MEMORY COLUMN STORE QUERYING THE IN-MEMORY COLUMN STORE IN-MEMORY JOINS AND AGGREGATION IN-MEMORY TRANSACTION MANAGER MANAGING OBJECTS IN THE IN-MEMORY COLUMN STORE APPENDIX A - WORKSHOP LOGISTICS LOGGING INTO THE ORACLE NETWORK (ORACLE OFFICE ONLY) ESTABLISHING A VPN CONNECTION TO THE VPN NETWORK LOGGING INTO AN ENTERPRISE MANAGER SESSION WORKSHOP LAB GUIDE CONVENTIONS WORKSHOP TIPS P a g e

3 Authors The Oracle Database In-Memory Workshop is based on the Oracle Database Hands-On Lab (August 2014) documentation and scripts authored by Maria Colgan (Master Product Manager) and Andy Rivenes (Senior Principal Product Manager). Content was also enhanced by Maria s comments during review. The additional content was developed in-house by the Oracle Solution Center with assistance by certain SC field organizations within Oracle (identified below in Acknowledgements). Any errors or issues found with the collateral or the accompanying VM Image should be communicated to the instructor of the Workshop. The key OSC authors and contributors are noted below: Geoff Grandstaff, OSC Solution Engineer, Oracle Solution Center Acknowledgements The OSC wants to thank the following individuals who contributed to the creation of this Workshop: Scott Black Eric Cohen Greg Day Antony Fewster Samir Kakli John Lang Murilo Nascimento Jim Neofotistos Jason Schwenneker Referenced Collateral for the Workshop Guide 1. Oracle Database Hands-On Lab (August 2014) authored by Maria Colgan and Andy Rivenes. 2. Star Schema Benchmark Revision 3, June 5, 2009 Pat O Neil, Betty O Neil, Xuedong Chen 3. Oracle Database 12c Grand Tour Workshop Student Workshop Lab Guide, William Summerhill and Michael B. Smith, Oracle Solution Center 4. Oracle Database In-Memory Data Sheet 5. Oracle Database Administrator's Guide 12c Release 1 (12.1) (E ) 6. Oracle Database Concepts 12c Release 1 (12.1) (E ) 7. Oracle Database Data Warehousing Guide 12c Release 1 (12.1) (E ) 8. Oracle Database Performance Tuning Guide 12c Release 1 (12.1) (E ) 9. Oracle Database Reference 12c Release 1 (12.1) (E ) 10. Getting started with Oracle Database In-Memory Part I - Installing & Enabling, July 28, 2014, Maria Colgan 3 P a g e

4 Disclaimer Unless explicitly identified as such, the sample code here is not certified or supported by Oracle; it is intended for educational or testing purposes only. The code samples may be modified but not redistributed. This document is intended for attendees of the workshop for their private use. It is not to be sold, distributed to others, or posted on internal or external web sites without the written consent of the Oracle Corporation. Copyright 2011, Oracle. All rights reserved. This documentation contains proprietary information of Oracle Corporation. It is provided under a license agreement containing restrictions on use and disclosure and is also protected by copyright law. Reverse engineering of the software is prohibited. If this documentation is delivered to a U.S. Government Agency of the Department of Defense, then it is delivered with Restricted Rights and the following legend is applicable: Restricted Rights Legend Use, duplication or disclosure by the Government is subject to restrictions for commercial computer software and shall be deemed to be Restricted Rights software under Federal law, as set forth in subparagraph (c)(1)(ii) of DFARS , Rights in Technical Data and Computer Software (October 1988). This material or any portion of it may not be copied in any form or by any means without the express prior written permission of Oracle Corporation. Any other copying is a violation of copyright law and may result in civil and/or criminal penalties. If this documentation is delivered to a U.S. Government Agency not within the Department of Defense, then it is delivered with Restricted Rights, as defined in FAR , Rights in Data-General, including Alternate III (June 1987). The information in this document is subject to change without notice. If you find any problems in the documentation, please report them in writing to the Oracle Solution Center, osc_us@oracle.com Oracle Corporation does not warrant that this document is error-free. Oracle and all references to Oracle Products are trademarks or registered trademarks of Oracle Corporation. All other products or company names are used for identification purposes only, and may be trademarks of their respective owners. 4 P a g e

5 1. WORKSHOP OVERVIEW This Workshop is designed to introduce you to Oracle Database In-Memory through a series of Hands-On Lab Exercises. Below is a list of the Labs that will be covered in this Workshop. 1. Monitoring and Querying the In-Memory Column Store Lab Setup and Monitoring the In-Memory Column Querying the In-Memory Column Store 2. In-Memory Joins and Aggregation 3. In-Memory Transaction Manager 4. Managing Objects in the In-Memory Column Store 5 P a g e

6 1.1. WORKSHOP ENVIRONMENT SETUP You will receive your own Virtual Machine environment complete with a Database 12c installation and an Enterprise Manager Cloud Control 12c R4 environment. All lab exercises will be run within your VM environment. Your instructor will provide you with your IP address during the class. USER NAMES AND PASSWORDS The table below contains a list of the usernames and passwords for the VM environment. Account Username Password Unix user oracle oracle12 SYS DB user sys oracle12 SYSTEM DB user system oracle12 EM12c user sysman oracle12 SSB user ssb oracle12 SQL SCRIPTS To reduce the amount of typing that you have to do during the labs, SQL*Plus scripts have been created to issue the commands during the lab exercises. When logging into SQL*Plus, change to the Scripts directory before invoking sql*plus so that the scripts can be called properly. 1. Script Location: /home/oracle/in_memory/sql WORKSHOP TIPS Here are a few tips and best practices for working with the Database In-Memory Workshop. 1. Using SSH: You can use any ssh utility (putty, term, etc) to ssh to your VM image. An assumption is being made in this workshop that you know how to ssh to a remote server so no instructions have been given. If you have any problems using ssh, please talk with your instructor. 2. Before invoking SQL*Plus, make sure you cd to the scripts directory (/home/oracle/in_memory/sql). All instructions assume that you invoked SQL*Plus from the scripts directory. SQL*Plus won t find the script if you didn t invoke it from the scripts directory. There is an alias called golab that you can invoke to quickly take you to the scripts directory. Also, there is an alias called gossb that you can invoke to easily log in as the ssb user. SSB SCHEMA The following five table star schema shown has been created under the ssb user in the pluggable database (PDB) imcs1. This star schema comes from the Star Schema Benchmark 1 designed by Pat O Neil from the department of Math and Computer Science in the University of Massachusetts. It has been modified from the TPC-H Benchmark, where the LINEITEM and ORDERS tables have been merged into one sales fact table called LINEORDER. The schema has been created with a 50GB scales factor. 1 Star Schema Benchmark Revision 3, June 5, 2009 Pat O Neil, Betty O Neil, Xuedong Chen 6 P a g e

7 MULTITENANT 2 A brief explanation of Oracle Multitenant. Oracle Multitenant is a new option for Oracle Database 12c Enterprise Edition that helps customers reduce IT costs by simplifying consolidation, provisioning, upgrades, and more. The multitenant architecture enables an Oracle database to function as a multitenant container database (CDB) that includes zero, one, or many customer-created pluggable databases (PDBs). A PDB is a portable collection of schemas, schema objects, and non-schema objects that appears to an Oracle Net client as a non-cdb. All Oracle databases before Oracle Database 12c were non-cdbs. A CDB includes the following components. Each of these components is called a container. Each container has a unique container ID and name within a CDB. Root (CDB$ROOT): The root, named CDB$ROOT, stores Oracle-supplied metadata and common users. An example of metadata is the source code for Oracle-supplied PL/SQL packages. A common user is a database user known in every container. A CDB has exactly one root. 2 Oracle Database 12c Grand Tour Workshop Student Workshop Lab Guide, William Summerhill and Michael B. Smith, Oracle Solution Center 7 P a g e

8 Seed (PDB$SEED): The seed, named PDB$SEED, is a template that you can use to create new PDBs. You cannot add objects to or modify objects in the seed. A CDB has exactly one seed. PDBs: A PDB appears to users and applications as if it were a non-cdb. For example, a PDB can contain the data and code required to support a specific application. A PDB is fully backward compatible with Oracle Database releases before Oracle Database 12c. The list below explains some of the properties of a PDB and CDB within a Multitenant architecture. A single CDB may contain many PDBs. The CDB and all its PDBs are at the same Oracle Database version. The CDB has the data dictionary for the Oracle-supplied system (in the root) and is shared as read-only across all the PDBs. Each PDB has its own private data dictionary for customer-created database objects. From a client perspective a PDB is backwards compatible with a pre-12.1 database. The PDB is the database. No changes to client code or database objects are necessary. A session sees only the single PDB it is connected to. A PDB can be unplugged from one CDB and plugged into another CDB. This operation can be done across different Oracle Database versions. A PDB can be cloned, either within a CDB or into another CDB. Operations on PDBs (creating, unplugging, plugging in, cloning, dropping, and setting the Open_Mode) are implemented as SQL statements accessible to the CDB administrator when connected to the CDB root Pluggable database functionality is fully interoperable with all database options. In a RAC database, each instance opens the CDB as a whole. Data Guard acts on the CDB as a whole. RMAN scheduled incremental, online backups are done on the CDB as a whole. A PDB can be selected for point-in-time recovery (PDB-PITR) while the other PDBs remain open. Your VM Image contains the following: A container database called imcs imcs contains a PDB called inmem1 8 P a g e

9 2. MONITORING AND QUERYING THE IN-MEMORY COLUMN STORE The following labs will guide you through the basic configuration and monitoring of the In-Memory column store. The labs will also highlight the key features of the In-Memory column store (IM column store): In-Memory Column Tables In-Memory Joins and Aggregation In-Memory Column Transaction Manager Dual row and column formats are a unique feature of Oracle Database In-Memory. In order to demonstrate these benefits we must first establish a performance baseline. It would be unfair to compare the IM column store with disk accessed data. After all, memory access in general is 10X faster than disk. In order to do a fair comparison the performance baseline will be established using the row store memory space, the buffer cache. For the purposes of the lab the database environment has been sized so that the tables used will fit in both the row store and the column store. The idea is that no buffer accesses will cause physical I/O. 9 P a g e

10 2.1. LAB SETUP AND MONITORING THE IN-MEMORY COLUMN STORE Oracle Database In-Memory The focus of this lab is to setup the lab environment and to demonstrate how to monitor the different parts of the In- Memory column store (IM column store). The lab is running in an Oracle VM environment using MB for the base memory and 2 processors. 10 P a g e

11 1. ssh to the VM Image you have been assigned (the IP address) and log in as the Linux user oracle (password is oracle12). 2. Linux user oracle s environment is already set up for the imcs container database. 3. Change directory to the Script Location /home/oracle/in_memory/sql (or you can use the alias golab). 4. Open the PDB inmem1 which we will use during the labs. select name, open_mode from v$pdbs; alter pluggable database inmem1 open; 11 P a g e

12 5. Check to see how the memory is currently allocated in the database by issuing the following commands: show sga (or you can use the script 1_01_show_sga.sql) show parameter inmemory_size (or you can use the script 1_02_01_show_inmemory.sql) show parameter sga_target The parameter inmemory_size sets the size of the IM column store. The default value is 0 which means that the IM column store is not used. The minimum setting is 100M when setting inmemory_size to a non-zero value. In a multitenant environment the setting for this environment in the root (CDB$ROOT) is the setting for the entire multitenant container database (CDB). This parameter can also be set in each pluggable database (PDB) to limit the maximum size of the IM column store for each PDB. The sum of the PDB values can be less than, equal to, or greater than the CDB value. However, the CDB value is the maximum amount of memory available in the IM column store for the entire CDB, including the root and all of the PDBs. Unless this parameter is specifically set for a PDB, the PDB inherits the CDB value, which means that the PDB can use all of the available IM column store for the CDB. 3 3 Oracle Database Administrator's Guide 12c Release 1 (12.1) (E ) 12 P a g e

13 6. ALTER SYSTEM SET sga_target = 7G scope=spfile; 7. There are only a few steps to set up the IM column store. The database compatibility must be set to or higher. Set inmemory_size to a non-zero value (minimum value 100M). The database must be restarted to initialize the IM column store in the SGA. The IM column store is a static pool in the SGA. SHOW PARAMETER compatible ALTER SYSTEM SET inmemory_size = 2G scope=spfile; 8. Restart the database to initialize the IM column store in the SGA. shutdown immediate; startup; alter pluggable database inmem1 open; 13 P a g e

14 9. show parameter inmemory (or you can use the script 1_02_01_show_inmemory.sql) Let s take a look at the additional In-Memory initialization parameters: INMEMORY_CLAUSE_DEFAULT allows you to specify a default mode for in-memory tables, for example it can be used to mark all new tables in memory or to force certain in-memory options by default if not explicitly specified in the syntax. The default value is an empty string, which means that only explicitly specified tables are populated into the IM column store. This is the same as setting this parameter to NO INMEMORY. If you want all new tables to be in memory, set INMEMORY_CLAUSE_DEFAULT to "INMEMORY". 14 P a g e

15 INMEMORY_FORCE By default any object with the INMEMORY attribute specified on it is a candidate to be populated into the IM column store. However, if INMEMORY_FORCE is set to OFF, then even if the in-memory area is configured, no tables are put in memory. The default value is DEFAULT. INMEMORY_MAX_POPULATE_SERVERS controls the maximum number of worker processes that can be started and is set to 0.5 X CPU_COUNT by default. INMEMORY_QUERY allows you to enable and the use of the IM column store either at the session or system level. The default value is ENABLE. INMEMORY_TRICKLE_REPOPULATE_PERCENT controls the maximum percentage of time that worker processes can perform trickle repopulation. The value of this parameter is a percentage of the INMEMORY_MAX_POPULATE_SERVERS parameter. OPTIMIZER_INMEMORY_AWARE enables or disables all of the optimizer cost model enhancements for inmemory. Setting the parameter to false causes the optimizer to ignore the in-memory property of tables during the optimization of SQL statements. This behavior can also be achieved by setting the OPTIMIZER_FEATURES_ENABLE initialization parameter to values lower than Now you have set up an In-Memory column store but Oracle Database In-Memory option is still not in use because no database objects have been populated into the IM column 11. This can also be verified by checking the feature tracking usages of the In-Memory column store. 4 Oracle Database Reference 12c Release 1 (12.1) (E ) 5 Getting started with Oracle Database In-Memory Part I - Installing & Enabling, July 28, 2014, Maria Colgan 15 P a g e

16 12. Login to Enterprise Manager Cloud Control 12c R4 as sysman using a browser. Confirm Certificate Secutity Exception. If using Chrome: 16 P a g e

17 If using Firefox: If using Internet Explorer: 17 P a g e

18 Now, login as sysman. 13. Targets -> Databases 18 P a g e

19 14. Expand the imcs Container Database and select the INMEM1 Pluggable Database. It is listed in the format cdbname_pdbname. 15. In the top left corner of your browser notice you are looking at the Home page for INMEM1 PDB. 19 P a g e

20 16. Let s take a look at In-Memory Central. Administration -> In-Memory Central 17. Login using the Named Credentials that are already populated. 18. You can see that the In-Memory column store is set up but no database objects are populated into the IM column store. 20 P a g e

21 19. The In-Memory area is sub-divided into two pools: a 1MB pool used to store the actual column formatted data populated into memory, and a 64K pool used to store metadata about the objects that are populated into the IM column store. The amount of available memory in each pool is visible in the V$INMEMORY_AREA view. The relative size of the two pools is determined by internal heuristics, the majority of the In-Memory area memory is allocated to the 1MB pool. To see the total IM column store usage query the V$INMEMORY_AREA performance view. SELECT * FROM v$inmemory_area; (or you can use the script 1_02_02_im_usage.sql) 21 P a g e

22 20. Connect as the ssb user in the inmem1 PDB. connect (if you are still in a SQL*Plus session) As mentioned in the Workshop Tips section, there is an alias called gossb that you can invoke to easily log in as the ssb user from a Linux shell. 21. To add objects to the IM column store the INMEMORY attribute must be set for the object. Four new columns INMEMORY, INMEMORY_PRIORITY, INMEMORY_DISTRIBUTE, and INMEMORY_COMPRESSION- have been added to the *_TABLES views to indicate the current In-Memory attributes for a table. Look at some of the In-Memory attributes of the current tables in the ssb schema by using the script Notice that each of these tables has been marked as CACHE. This means that they should be cached in the buffer cache. Also note that none of the tables are currently marked for the In-Memory Column Store. 22. Mark all of the current tables in the ssb schema for the In-Memory Column Store. ALTER TABLE <table name> INMEMORY; (or you can use the script 1_05_im_set_attr.sql) 22 P a g e

23 INMEMORY_PRIORITY INMEMORY: This attribute indicates whether this table should be populated in the IM column store. The values are ENABLED or DISABLED. INMEMORY_PRIORITY: When the object should be populated is controlled by INMEMORY_PRIORITY. By default it is set to NONE, which means Oracle automatically decides when to populate the table into the IM column store. This is also referred to as on demand as Oracle typically populates the table after it has been accessed for the first time. Alternatively, the priority level can be set, which queues the population of the table into the IM column store immediately. The queue is drained from CRITICAL to LOW priority (CRITICAL -> HIGH-> MEDIUM -> LOW). This is also referred to as at startup as Oracle populates the IM column store using this prioritized list each time the database starts. INMEMORY_DISTRIBUTE: Each node in a RAC environment has its own IM column store. It is possible to have completely different objects populated on each node, or to have larger objects distributed across all of the IM column stores in the cluster. It is also possible to have the same objects appear in the IM column store on every node (Engineered Systems only). How each object should populate in the column store is displayed in the INMEMORY_DISTRIBUTE column of the *_TABLES views. The default value is AUTO, where Oracle automatically decides how an objects should be distribute among the IM column store in a RAC environment. On a single instance database this attribute has no effect. 23 P a g e

24 INMEMORY_COMPRESSION: Each object that is brought into the IM column store is compressed. Oracle offers multiple compression techniques which provide different levels of compress and performance. By default, data is compressed using the FOR QUERY option. This provides the best balance between compression and performance. 24. Although each of the tables now has its in-memory attribute set, the tables still don t reside in the In-Memory Column Store. Remember, by default the In-Memory Column Store is only populated when the object is accessed. Query v$im_segments 25. Execute the following select count(*) query for each table to trigger the population (use the 1_05_im_start_pop.sql 24 P a g e

25 When these select statements are run background processes populate the IM column store. These processes are called worker processes and come in the form of ora_w***_imcs. It is possible to monitor the progress of the population by querying the view v$im_segments. 26. Remember the view V$IM_SEGMENTS shows what objects are in the column store. It also contains information about the population of those objects. Run the 1_06_im_populated.sql script to see the current population status for the objects in the IM column 25 P a g e

26 Having 0 (zero) in the BYTES_NOT_POPULATED column indicates that the entire table has been completely populated into the IM column store. Don t move on from this step until all 5 tables have been fully populated. What about compression? How large are the actual objects stored in the IM column store? Let s take a look. 27. The actual size of the objects on disk and their size in the IM column store can also be seen in the V$IM_SEGMENTS performance view. Using this information we can calculate the compression ratio for each of the objects populated in IM column store. Note, if the objects were compressed on disk this query would not yield the correct compression ratio. Run the 1_07_comp_ratio.sql script to see the object sizes in the column 26 P a g e

27 NOTE: this query assumes that the tables are not compressed on disk. That is not true for one of our tables LINEORDER. LINEORDER is compressed on disk using basic compression so the compression ratio on LINEORDER should actually be 2.4X based on its uncompressed size. 28. Now that the IM column store has been populated with some tables, let s take a look at In-Memory Central in Enterprise Manager Cloud Control 12c. This time, we will navigate to In-Memory Central by first going to imcs CDB Home page. Go to Favorites -> imcs (Database Instance) -> imcs. 29. Administration -> In-Memory Central 27 P a g e

28 30. Click the Search icon to select the imcs_inmem1 PDB (if Database Login is necessary). 28 P a g e

29 29 P a g e Oracle Database In-Memory

30 As you see in the image above, make sure you are in the INMEM1 PDB. If not, switch to the INMEM1 PDB as show in the image below. 31. Review the In-Memory Objects Size Map for all five tables. 30 P a g e

31 31 P a g e Oracle Database In-Memory

32 32. View the Compression Factor in the In-Memory Populated Object Statistics section. Also note that the In-Memory Populated Object Statistics is grouped by INMEMORY_PRIORITY (and also the Total). As we noted when you earlier, by default INMEMORY_PRIORITY is set to NONE, which means Oracle automatically decides when to populate the table into the IM column store. In Lab 5, you will assign INMEMORY_PRIORITY to some partitions and then you will see those also listed in In-Memory Populated Object Statistics (CRITICAL -> HIGH-> MEDIUM -> LOW). 32 P a g e

33 33. View In-Memory Enabled Object Statistics. 34. View In-Memory Objects Distribution. 35. View the data populated in the In-Memory Objects Search. SQL scripts that were executed earlier in the lab have also show much of this information. Conclusion In this first part of this lab you should have seen how simple it is to configure the IM column store by setting the initialization parameter INMEMORY_SIZE. Remember that the IM column store is a new static pool in the SGA that cannot be resized dynamically and is not managed via the automatic SGA algorithm. 33 P a g e

34 You also had an opportunity to view objects in the IM column store and to see how much space they use. In this lab you populated the five tables in the ssb schema into the IM column store. Remember the population speed depends on the CPU capacity of the system as the data compression is a CPU intensive operation. The more CPU you have the faster the populations will occur. Then you got to see exactly what our new In-Memory compression algorithms are capable of. Remember the goal of the algorithms is not space reduction as with our on-disk compression but performance. The compression ratio is dependant of the data distribution within each column. Finally, you were introduced to In-Memory Central in Enterprise Manager Cloud Control 12c. 34 P a g e

35 2.2. QUERYING THE IN-MEMORY COLUMN STORE This lab executes a series of queries against the large fact table LINEORDER, in both the buffer cache and the IM column store, to demonstrate the different ways the IM column store can improve query performance above and beyond the basic performance benefits of accessing data in memory only. 35 P a g e

36 1. Let s begin with the simplest of queries, "What is the most expensive order we have received to date?" Since we have no alternative access paths (indexes or views) the execution plan for this query is a full table scan of the LINEORDER table. First, execute the query against the IM column store type using the 2_01_im_query.sql 2. To execute the same query against the buffer cache we will need to disable the IM column store via a hint called NO_INMEMORY, as the Optimizer will favor accessing the data in the column store when the execution plan is a full table scan, as it is in this case. Execute the query against the buffer cache using the 2_02_buffer_query.sql As you can see the query executed extremely quickly in both cases because this is purely an in-memory scan. Also, these labs are for learning the concepts as they are hosted on small VM and not a larger scale. However, the performance of the query against the IM column store was significantly faster than on the traditional buffer cache - why? 36 P a g e

37 The IM column store only has to scan a single column - lo_ordtotalprice - while the row store has to scan all of the columns in each of the rows until it reaches the lo_ordtotalprice column. The IM column store also benefits from the fact the data is compressed so the volume of data scanned is less. Finally the column format requires no additional manipulation for SIMD vector processing (Single Instruction processing Multiple Data values). Instead of evaluating each entry in the column one at a time, SIMD vector processing allows a set of column values to be evaluated together in a single CPU instruction. 3. How can you tell if your query is using the IM column store Table? Run the script 2_03_im_xplan.sql. Then, run the 37 P a g e

38 You will notice that both of the execution plans are similar but the first query run against the IM column store has a new set of keywords "IN MEMORY", while the buffer cache query does not. These keywords indicate that the LINEORDER table has been marked for IN MEMORY and we may use the column store in this query. What do we mean by "may use"? There are a small number of cases were we won t use the IM column store even though the object has been marked IN MEMORY. This is similar to how the keyword STORAGE is used on Exadata environments. 4. In order to confirm that the IM column store was used, we need to examine the session level statistics. You can monitor the session level statistics by querying the performance views v$mystat and v$statname. All of the statistics related to the IM column store begin with IM. You can see the full list of these statistics by executing the SQL script called 2_05_list_im_stats.sql. This script exits SQL*Plus at the end. The reason is to clear the session statistics before the next lab step. Remember, you can execute the alias gossb to connect back to SQL*Plus as the ssb 5. That's a lot of statistics. We are just interested in understanding if our query really executed in the IM column store. Let's examine the following statistic: IM scan CUs columns accessed: Total number of Compression Units (CU) actually accessed by this query. 38 P a g e

39 As our query did a full table scan of the LINEORDER table, all of the CUs in the LINEORDER table were actually accessed. Execute the script 2_06_im_stats.sql. This script exits SQL*Plus at the end. The reason is to clear the session statistics before the next lab step. Remember, you can execute the alias gossb to connect back to SQL*Plus as the ssb user. A few other scripts also exit SQL*Plus but just remember to use the gossb This statistic shows that the LINEORDER table has several CUs in the In-Memory Column Store. 6. Now execute the same steps for the buffer cache query by executing the script 39 P a g e

40 This time you will notice that the IM statistics never get incremented because the query bypassed the IM column store and executed on the buffer cache instead. 7. Traditionally, a full table scan is not the most efficient execution plan when you are looking for a specific entry in a table. Tables stored in the IM column store breaks this stereotype and we can demonstrate this by looking for a specific order in the LINEORDER table based on the orderkey. Execute the query against the IM column store type by running the script 40 P a g e

41 8. Execute the same query against the buffer cache type by running the script 41 P a g e Again the IM column store shows an impressive improvement over the traditional buffer cache - but how? Remember the IM column store has access to a storage index on each of the columns, which enables it to do min/max pruning. The where clause predicate is compared to the min/max range for each in-memory segment of the corresponding column, and if the value doesn t fall in the specified range then the segment is skipped completely. 9. You can tell that min/max pruning occurred by looking at three of the IM session statistics. IM scan CUs columns accessed IM scan segments minmax eligible

42 IM scan CUs pruned Execute the script 42 P a g e

43 From these results you can see just how effective the min/max pruning can be. The statistics show that a large number of segments were pruned out because no rows passed the min/max comparison. That means we never scan those segments. Only a small number of segments had a min / max range that our value fell into (scan CUs columns accessed). 10. At this stage you might be thinking that a simple index on the lo_orderkey would provide the same performance as the IM column store and you would right. There is an invisible index already created on the lo_orderkey column of the LINEORDER table. By using the parameter OPTIMIZER_USE_INVISIBLE_INDEXES we can compare the performance of the IM column store and the index. Execute the script 2_11_index_comparison.sql to see just how well the index performance against the column 43 P a g e

44 11. However, it's not often that an analytical query has only one equality where clause predicate. What happens when there are multiple single column predicates on a table? Traditionally you would create a multi-column index. Can storage indexes compete with that? Let's change our query to look for a specific line item in an order and monitor the session statistics. Execute the script 44 P a g e

45 Based on the elapsed time and the session statistics you can see that the In-Memory storage index is still used. In fact, we are able to use multiple storage indexes together in a similar manner to how Oracle Database can combine multiple bitmap indexes. 45 P a g e

46 12. There isn't a multi-column index created for this query so running it against the buffer cache means we must do a full table scan. You can see how that affects performance by executing the script 13. What if the where clause predicate isn t an equality predicate? Can you still benefit from min/max pruning? Let's change the query to ask "What was the most expensive bulk order to fill, where a bulk order is one that has a quantity greater than 52 items?" Execute the script 46 P a g e

47 14. To execute the same query against the buffer cache we will need to disable the IM column store via a hint, as the Optimizer will favor accessing the data in the column store when the execution plan is a full table scan, as it is in this case. To execute the query against the buffer cache type run the SQL script 2_16_min_max_buffer.sql. 47 P a g

48 The IM column store still out-performs the buffer cache because it only has to scan a single column - lo_quantity - and although the where clause predicate is not an equality, we are still able to use the In-Memory storage index. 15. Let's kick it up a notch and see what happens when we have more complex where clause predicates that include multiple columns and a subselect on the LINEORDER table. The query this time is to determine which of the expensive bulk orders generated the least amount of revenue for the company when shipped by truck versus air. Execute the script 48 P a g e

49 16. Execute the same query against the buffer cache type by running the script 49 P a g e

50 Even with the all of these complex predicates the IM column store query is still significantly faster, showing that for large scan operations it is the most efficient approach. Conclusion In this second part of the lab you had an opportunity to try out Oracle Database In-Memory with queries that run against a 23,996,604 row table (LINEORDER), which resides in both the IM column store and the buffer cache. From a very simple single row look-up, to more complex queries with multiple sub-queries, the IM column store was able to outperform the buffer cache queries. Remember both sets of queries are executing completely within memory, so that's quite an impressive improvement. These significant performance improvements are possible because of Oracle's unique In-Memory columnar format that allows us to only scan the columns we need and to take full advantage of SIMD vector processing. We also got a little help from our new In-Memory storage indexes, which allow us to prune out unnecessary data. Remember that with the IM column store, every column has a storage index that is automatically maintained for you. In this lesson you also got an opportunity to monitor an in-memory workload. Those of you familiar with Oracle's Exadata Engineered Systems will see a lot of similarity here. There is a new keyword (IN MEMORY) in the execution plan to indicate which operations within that plan are eligible to be executed on the IM column store. To confirm that these operations actually took place in the IM column store you need to examine the new set of In- Memory session statistics. 50 P a g e

51 3. IN-MEMORY JOINS AND AGGREGATION Up until now we have been focused on queries that scan only one table, LINEORDERS. Let's broaden the scope of our investigation to include joins and parallel execution. This lesson executes a series of queries that begin with a single join between the fact table, LINEORDER, and a dimension table but works up to a 5 table join. The queries will be executed in both the buffer cache and the column store, to demonstrate the different ways the column store can improve query performance above and beyond the basic performance benefits of scanning data in a columnar format. 51 P a g e

52 1. We will begin with the simplest join query, a join between the fact table LINEORDER and the dimension table DATE_DIM. This is a What if style query that calculates the amount of revenue increase that would have resulted from eliminating certain company-wide discounts in a given percentage range for products shipped on a given day (Christmas Eve 1996). Execute the query against the IM column store type by running the script 2. To execute the query against the buffer cache type run the script 52 P a g e

53 The IM column store has no problem executing a query with a join because it is able to take advantage of Bloom Filters. A bloom filter transforms a join to a filter that can be applied as part of the scan of the fact table. Bloom filters were originally introduced in Oracle Database 10g to enhance hash join performance and are not specific to Oracle Database In-Memory. However, they are very efficiently applied to columnar data via SIMD vector processing. Here is a brief description of how they work: When two tables are joined via a hash join, the first table (typically the smaller table) is scanned and the rows that satisfy the 'where' clause predicates (for that table) are used to create a hash table. During the hash table creation a bit vector or bloom filter is also created based on the join column. The bit vector is then sent as an additional predicate to the second table scan. After the where clause predicates have been applied to the second table scan, the resulting rows will have their join column hashed and it will be compared to values in the bit vector. If a match is found in the bit vector that row will be sent to the hash join. If no match is found then the row will be disregarded. On Exadata the bloom filter or bit vector is passed as an additional predicate so it will be offloaded to the storage cells making bloom filtering even more efficient. It is easy to identify bloom filters in the execution plan. They will appear in two places, at creation time and again when it is applied. Below is the plan for the query in step 1 with the bloom filter: You can also see what join condition was used to build the bloom filter by looking at the predicate information under the plan. Look for SYS_OP_BLOOM_FILTER in the filter predicates. 53 P a g e

54 3. To display the full execution plan for the query in step1 execute the script 54 P a g e

55 4. Let's try a more complex query that encompasses two joins and an aggregation of a lot more data. In this case the query will compute the revenue generated from a specific product in the busiest month of the year, December. As we are dealing with more data let's using parallel execution to speed up the elapsed times so we don't need to wait too long for the results. To execute the query against the IM column store type, run the script 5. To execute the query against the buffer cache type, run the script 55 P a g e

56 6. If you examine the execution plan for this SQL statement you will see two bloom filters were created. The bloom filter is created on the lo_partkey and the lo_orderdate column. Oracle Database is not limited to just one bloom filter per scan. It is possible to apply multiple bloom filters on a single table scan when appropriate. Execute the script 56 P a g e

57 7. Let's introduce a third table to our query. This time our query will compare the revenue for different product classes, from suppliers in a certain region for the year This query returns more data than the others we have looked at so far. To execute the query against the IM column store type, run the script 57 P a g e

58 8. To execute the query against the buffer cache type, run the script 58 P a g e

59 59 P a g e Oracle Database In-Memory

60 9. The IM column store continues to out-perform the buffer cache query but what is more interesting is the execution plan for this query. Let's take a look. View the execution plan for the query by executing the script 60 P a g e

61 In this case, we see three join filters have been created and applied to the scan of the LINEORDER table, one for the join to the PART table, one for the join to DATE_DIM, and one for the join to the SUPPLIER table. How is Oracle able to apply two join filters when the join order would imply that the LINEORDER is accessed before the SUPPLIER table? This is where Oracle s 30 years of database innovation kick in. By embedding the column store into Oracle Database we can take advantage of all of the optimizations that have been added to the database over the last 11 releases. In this case, the Optimizer has switched from its typically left deep tree to create a right deep tree using an optimization called 'swap_join_inputs'. What this means for the IM column store is that we are able to generate multiple bloom filters before we scan the necessary columns for the fact table, meaning we are able to benefit by eliminating rows during the scan rather than waiting for the join to do it. 10. To check just how much benefit the multiple bloom filters provide, execute 3_11_3join_im_stats.sql and observe the session 61 P a g e

62 62 P a g e Oracle Database In-Memory

63 63 P a g e Oracle Database In-Memory

64 11. Up until this point we have been focused on joins and how the IM column store can execute them incredibly efficiently. Let's now turn our attention to more OLAP style "What If" queries. In this case our query examines the yearly profits from a specific region and manufacturer over our complete data set. To execute the query against the IM column store type, run the script 64 P a g e

65 12. To execute the query against the buffer cache type, run the script 65 P a g e

66 66 P a g e Oracle Database In-Memory

67 Our query is more complex now and the majority of the time is no longer spent in the scan operations of the execution plan. More time is now being spent joining, sorting, and aggregating the data. You can check it out for yourself by examining the execution plan. Let s take a look in the next step. 13. To examine the execution plan for our query execute the script 67 P a g e

68 Let's go to the next step to see how an alternative execution plan for our query will help improve the performance of the operations that occur after the scan operations. 14. In order to speed up the operations above in the execution plan, Oracle has introduced a new Optimizer transformation, called Vector Group By 6. This transformation is a two-part process not dissimilar to that of star transformation. First, the dimension tables are scanned and any where clause predicates are applied. A new data structure called a key vector is created based on the results of these scans. The key vector is similar to a bloom filter as it allows the join predicates to be applied as additional filter predicates during the scan of the fact table, but it also enables us to conduct the group by or aggregation during the scan of the fact table instead of having to do it afterwards. 6 Oracle Database Data Warehousing Guide 12c Release 1 (12.1) (E ) 68 P a g e

69 The second part of the execution plan sees the results of the fact table scan being joined back to the temporary tables created as part of the scan of the dimension tables. These temporary tables contain the payload columns (columns need in the select list) from the dimension table. The combination of these two phases dramatically improves the efficiency of a multiple table join with complex aggregations. Both phases are visible in the execution plan of our query. Let s take a look at Vector Group By in action. This time we will expand the query to look at the profits we made across the 6 years we have been in business, for all customers. We will also change the comparison criteria on this step. Instead of comparing the IM column store performance to the buffer cache, we will compare the query performance with (in Step 16) and without (in Step 17) Vector Group By. 15. Execute the script 3_18_vgb2_im.sql with vector group Output condensed 16. Execute the script 3_19_vgb2_im_novgb.sql with vector group by disabled. 69 P a g e

70 @ 3_19_vgb2_im_novgb.sql Output condensed 17. Examine the Vector Group By execution plan by executing the script 70 P a g e

71 Output condensed (full explain plan shown in Lab Guide after Conclusion section of this lab) Conclusion In this Lab we saw our performance comparison expanded to queries with both joins and aggregations. You had an opportunity to see just how efficiently a join, which is automatically converted to a bloom filter, can be executed on the IM column store. You also got to see just how sophisticated the Oracle Optimizer has become over the last 30 years, when it used a combination of complex query transformations to find the optimal execution plan for a star query. Finally you got an opportunity to see the new vector group by transformation in action. This new transformation allows us to not only convert joins to filter applied as part of a scan but also to complete all aggregation operations as part of the scan of the fact table. 71 P a g e

72 72 P a g e Oracle Database In-Memory

73 73 P a g e Oracle Database In-Memory

74 4. IN-MEMORY TRANSACTION MANAGER It is clear that the IM column store can dramatically improve the performance of all types of queries but very few database environments are read only. For a column store to be truly integrated into a database environment it has to be able to handle both bulk data loads AND online transaction processing. This lesson demonstrates how the Oracle IM column store is the only in memory column store that can handle both bulk data loads and online transaction processing today. We will do this by running a 32-query workload concurrently with both a bulk load of 1 full day's worth of data and a more OLTP style DML test (single row DML). Our query workload is a mix of 7 different kinds of queries, starting from the most basic single column lookup, to a complex multi-table join with multiple aggregations. The where clause predicates for the queries begin simply with equality predicates but become more complete with less than and between predicates seen in the later queries. 74 P a g e

75 1. Before we begin our bulk load DML test we need to extract some data from the existing data set to act as the "new" data. We will do this by issuing two CREATE TABLE AS SELECT commands. One against the LINEORDER table to extract one day's worth of data, and one against the DATE_DIM table so we can add a new day. Below is example of one of the queries we will use: Here we generate a new day s worth of data in the LINEORDER table by adding a day to the two date columns (lo_orderdate + 1, lo_commitdate + 1). 2. Run the script 75 P a g e

76 3. You can confirm that this step completed successfully by checking for the presence of two new tables in the ssb schema. You can do this using the following query: One of these tables will be used to bulk load the LINEORDER table (ETL_LO) and the other to bulk load the DATE_DIM table (ETL_DD). Both tables need to be loaded together to maintain referential integrity. 4. Just one other piece of housekeeping to do before we can get started on the test. A results table needs to be created in the database so we can easily compare the elapsed time of the 32-query workload during each load test. To create the results table execute the script 76 P a g e

77 5. All good performance comparison tests require a baseline to be established and this is no exception. We will establish a baseline by executing the 32-query workload without any concurrent DML. The SQL script 4_02_step2.sql will automatically execute the workload and record the response time for each query in the new table LESSON4_RESULTS that was created in the previous step. To establish the baseline execute the script 77 P a g e

78 6. Now let's rerun the query workload while we also execute a bulk load into the LINEORDER table. Remember this load will add one extra day of orders to the LINEORDER table, approximately 3,000 rows. It will also add the corresponding day to the DATE_DIM dimension table. NOTE: With a bulk load all of the data is inserted into the table in a single transaction (only one commit). That means none of the data will be visible to the session doing the queries until the bulk load completes. This is the typical nature of a bulk load done in a data warehouse or an operational store. To kick off both the query workload and the bulk load DML scripts execute the script 4_03_bulk.sql. Note: This script will invoke a host 78 P a g e

79 79 P a g e Oracle Database In-Memory

80 80 P a g e Oracle Database In-Memory

81 7. The next part of our comparison will rerun the query workload while we also execute a set of more OLTP-style DML commands (INSERTS, UPDATES, DELETES) into the LINEORDER table. There are 26 DML commands in the workload (20 inserts, 3 updates, and 3 deletes). Each command commits immediately and none of the transactions are rolled back. NOTE: As each DML command commits immediately, the session executing the queries will see the information immediately after the commit. To kick off both the query workload and the OLTP style DML execute the script 4_04_single.sql. Note: This script will invoke a host 81 P a g e

82 82 P a g e Oracle Database In-Memory

83 83 P a g e Oracle Database In-Memory

84 84 P a g e Oracle Database In-Memory

85 8. With the three runs complete, let's compare the response time of each query in our workload in the three tests. We can do the comparison by selecting the response time for each query across our three-performance run: baseline, bulk DML, and single record DML. To run the comparison query execute the script 85 P a g e

86 Conclusion One of the most commonly asked questions about the new IM column store is, how is it impacted by DML. This Lab demonstrated just how little impact DML has on the performance of the IM column store. 86 P a g e

87 Remember, new data added as part of a bulk load operation is only visible after the session doing the DML commits. If the bulk load is done using a direct path operation, the data is written directly to disk and bypasses the buffer cache and the IM column store. The next query that accesses that data will trigger the newly inserted data to be populated into the IM column store. Single row change done via the buffer cache (OLTP style changes), are typically committed immediately and are reflected in the column store as they occur. The buffer cache and the column store are kept transactionally consistent. 87 P a g e

88 5. MANAGING OBJECTS IN THE IN-MEMORY COLUMN STORE During this lab we will cover some other management capabilities for the IM column store. 88 P a g e

89 1. Create a list partitioned table using the script 5_01_customer_region.sql 89 P a g e

90 2. Mark the partition America to be populated in the IM column store and start the population. Execute the script 5_02_part.sql 90 P a g e

91 3. Check for the partition in In-Memory Central. 4. In the IM column store, data can be compressed, and SQL queries execute directly on compressed data. 7 The IM column store compression methods are as follows: NO MEMCOMPRESS MEMCOMPRESS FOR DML MEMCOMPRESS FOR QUERY LOW MEMCOMPRESS FOR QUERY HIGH MEMCOMPRESS FOR CAPACITY LOW 7 Oracle Database Administrator's Guide 12c Release 1 (12.1) (E ) 91 P a g e

92 MEMCOMPRESS FOR CAPACITY HIGH In this step, add the partition Asia to the IM column store with a priority of medium and MEMCOMPRESS FOR QUERY HIGH. Also populate the data into the IM column store. Use the script 5. Check for the partition in In-Memory Central. 92 P a g e

93 6. Query USER_TAB_PARTITIONS using the script 7. Disable the Asia and America partitions for the IM column store using the NO INMEMORY clause. Then query USER_TAB_PARTITIONS again. Use the scripts 5_05_no_part.sql and 5_04_part_attributes.sql. 93 P a g e

94 8. Enable the table CUSTOMER_REGION for the IM column store and start the population process. Then query USER_TAB_PARTITIONS again. Use the scripts 5_06_all_part.sql 94 P a g e

95 9. Check for the partitions in In-Memory Central. 95 P a g e

96 10. Execute the script 5_07_alter.sql to change the IM column store priority and compression method of some of the partitions. Then query USER_TAB_PARTITIONS using the 96 P a g e

97 11. Check for the partitions in In-Memory Central. Also note the Priority column for the In-Memory Populated Object Statistics. 12. Just like earlier in the Lab, disable the Asia and America partitions for the IM column store using the NO INMEMORY clause. Then query USER_TAB_PARTITIONS again. Use the scripts 5_05_no_part.sql 97 P a g e

98 13. Check for the partitions in In-Memory Central. 98 P a g e

99 14. Disable the table CUSTOMER_REGIONS for the IM column store using the NO INMEMORY clause. Then query USER_TAB_PARTITIONS again. Use the scripts 5_08_no.sql Conclusion In this Lab we covered some management capabilities for the IM column store. 99 P a g e

100 6. APPENDIX A - WORKSHOP LOGISTICS Before you begin the workshop, there are a few things you must understand in order to work through it successfully. Your instructor will more than likely guide you through these tasks, but they are here for your reference. If this is your first workshop, we recommend that you review this section. If you ve taken a Workshop from us before and are aware of these Logistics, then feel free to skip ahead to Section 2 Workshop Overview. 1. Logging into the Oracle WiFi Network (Oracle Office Only) 2. Establishing a Secure VPN Connection to the Oracle Internal Network 3. Logging into an Enterprise Manager Session (if applicable) 4. Workshop Lab Guide Conventions 5. Workshop Guidelines 6.1. LOGGING INTO THE ORACLE NETWORK (ORACLE OFFICE ONLY) Welcome to the Oracle office. If you are a visitor in our office, you must first log on to our clear-guest network before you can access the OSC workshop network. Follow the instructions below. 1. Connect to the clear-guest network 100 P a g e

101 2. Launch Internet Explorer and go to google.com. You will be redirected to the following login screen. Userid: guest Password: <Supplied by Instructor> Click Submit If you were able to login successfully, you should see the google.com page ESTABLISHING A VPN CONNECTION TO THE VPN NETWORK If you have network access through your work location, or you are successfully on Oracle s clear-guest network, follow the steps below. 1. To access the SSL-VPN workshop network connect to the following URL with Internet Explorer 6.0 or above USER ID: Provided by the instructor Password: Provided by the instructor Click Sign In 101 P a g e

102 2. Click on the L3 connection link. 3. Accept the Active X control prompt. 102 P a g e

103 4. Click Install to install the Array Networks control 5. The install process will take a minute or so. Be patient. 103 P a g e

104 6. When the IP address shows, you are on the workshop network 104 P a g e