SEIZE THE DATA SEIZE THE DATA. 2015

Size: px

Start display at page:

Download "SEIZE THE DATA SEIZE THE DATA. 2015"

Angelica Little
5 years ago
Views:

2 Vertica Resource Management Ramesh Narayanan, Vertica Professional Services August 10, 2015

4 Agenda What are we going to talk about? Resource Manager Resource Pools Resource Pool Parameters Monitoring Resource Pools 4 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Copyright 2015 Hewlett-Packard Development

5 Why Have a Resource Manager? System table queries DBD Recovery Ad hoc queries Vertica Cluster Refresh Web Apps Tuple Mover (moveout/mergeout) Reporting Tools Loads 5 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

6 RESOURCE MANAGER Query Execution Lifecycle Create Query Plan on Initiator Reject if request requires resources beyond availability Distribute Query Plan to Executors Create Local Query Plan on Executors Determine required resources Determine admission to Executors Run query Queue query - possible reject Launch Execution Engine on Executors Manage runtime resources Process results and send to initiator node for aggregation 6 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

8 Resource Pools How do we manage mixed workload in Vertica? Control memory allocation, concurrency, queue timeout, and priority Control user queries and internal operations Tuned to meet different concurrency requirements User-defined pools created to reserve and prioritize system resources Users are assigned only one pool 8 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

9 Vertica Memory Allocation GENERAL Pool - 95% OS - 5% By default the General Pool gets 95% of the total system memory The remaining 5% is used by the Operating System and Vertica's catalog 9 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

10 Built-In Resource Pools GENERAL 95% of RAM 64M SYSQUERY 100M SYSDATA 200M TM (Tuple Mover) WOSDATA JVM RECOVERY REFRESH DBD Reserved Unreserved 10 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

11 Built-in Pools (1 of 2) General Default pool where user queries execute Other pools can borrow its memory to satisfy requests Capped as a percentage of system memory Sysquery System table queries use this pool during execution Sysdata System table results are stored in the pool s memory space TM Used for Tuple Mover tasks (moveout and mergeout) 11 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

12 Built-in Pools (2 of 2) WOSDAT Used to load data 2G or 25% of available memory If total RAM per node > 20G, 2G is reserved Minimize memory size if only performing bulk loads JVM Dedicated for Java UDx 2G or 10% of available memory DBD Used to control resource usage for DBD internal processing RECOVERY Used for handling node recovery (scratch and buddy recovery) REFRESH Used to allocate resources for projection refreshes 12 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

14 Resource Pool Parameters (1 of 6) MEMORYSIZE reserved memory (set to 0 for unreserved) MAXMEMORYSIZE caps memory usage available even when borrowing from GENERAL pool 14 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

15 Resource Pool Parameters (2 of 6) MAXCONCURRENCY maximum number of concurrent queries PLANNEDCONCURRENCY Expected number of queries running against this pool 15 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

16 Resource Pool Parameters (3 of 6) EXECUTIONPARALLELISM limits the number of threads used to process any single query Query Threads Results PRIORITY Specifies order in which Resource Manager reserves memory 16 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Resource Pool Parameters (4 of 6) RUNTIMEPRIORITYTHRESHOLD specifies the execution time (in seconds) before the Resource Manager assigns RUNTIMEPRIORITY RUNTIMEPRIORITY indicates the amount of

17 Resource Pool Parameters (4 of 6) RUNTIMEPRIORITYTHRESHOLD specifies the execution time (in seconds) before the Resource Manager assigns RUNTIMEPRIORITY RUNTIMEPRIORITY indicates the amount of resources (CPU, I/O bandwidth) dedicated to running queries in the resource pool 17 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

18 Resource Pool Parameters (5 of 6) RUNTIMECAP maximum amount of time any query on the pool can execute Also a parameter for individual users to limit query duration by specific users QUEUETIMEOUT maximum amount of queued time for a given query CASCADE TO Indicates a secondary resource pool used for queries that exceed the RUNTIMECAP of their assigned pool 18 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

19 Resource Pool Parameters (6 of 6) CPUAFFINITYMODE mode in which CPU affinity operates for this resource pool any (default), exclusive or shared CPUAFFINITYSET set of CPUs on which queries associated with this pool are executed These parameters should be changed only after consulting with Vertica technical support 19 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

20 Resource Pool Parameters PLANNEDCONCURRENCY Scenario: 6 users are assigned to a pool named "ABC" and all users are running 1G queries simultaneously What pools will the memory come from? ABC pool parameters: Query 1 1G MEMORYSIZE 4G PLANNEDCONCURRENCY 4 MAXCONCURRENCY 5 Query 2 Query 3 Query 4 1G 1G 1G ABC Allotted memory size per query So in this case 4G 4 = [MAX]MEMORYSIZE PLANNEDCONCURRENCY = 1G per query Query 5 Query 6 1G 1G General Queue 20 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

21 Resource Pool Defaults Parameter default description is_internal f the pool is not built-in memorysize null no reserved memory maxmemorysize null no maximum memory executionparallelism auto defaults to # of cores per node priority 0 no priority on queued transactions runtimepriority medium medium priority once threshold is met runtimeprioritythreshold 2 transaction runs at high priority for 2 seconds queuetimeout 300 transaction will time out if queued for 5 minutes plannedconcurrency auto defaults to # of cores per node maxconcurrency null unlimited concurrency runtimecap null no transaction time limit (allows run-away queries) singleinitiator f for backwards compatibility only; retain default cpuaffinityset null Set to run all queries on any CPU cpuaffinitymode Any Queries from this pool can run on any CPU 21 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

22 Reject versus Queue Reject query when too much memory per node is required OR the query waits in queue longer than the timeout parameter Timeout default = 5 minutes, configurable Shown as ABORT in resource_rejections system table Queue query when memory requirements cannot be met with current and planned memory allocation Shown as WAIT in resource_queues system table 22 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

23 User-Defined Pools Define your own specialized pools All parameters are customizable If queries need additional resources they can pull from the General pool Can spill into general pool (since MAXMEMORY is not specified) with a relative priority of 100 CREATE RESOURCE POOL load_pool MEMORYSIZE '10G' PRIORITY 100; User-defined pool name 10 GB reserved memory 23 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

24 User Rules User queries run in General or SYSQUERY pools unless a user-defined pool is specified GENERAL/ SYSQUERY Userdefined/ SYSQUERY A user is only assigned to 1 pool Groups cannot be assigned to a pool GENERAL/ SYSQUERY Userdefined GENERAL/ SYSQUERY Admin group 24 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Use Case Web Application for Cell Provider The client has a web application with an interactive portal for monthly usage reports When IT is running batch reports, the web application sometimes takes

25 Use Case Web Application for Cell Provider The client has a web application with an interactive portal for monthly usage reports When IT is running batch reports, the web application sometimes takes a long time to refresh and end users complain How can we provide a better user experience to web application users without stopping batch reports? 25 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

26 Use Case Web Application for Cell Provider Answer Create a pool for the web page refresh queries Associate this pool with the database user that runs the web application queries Create a pool to limit the batch reports to a fixed memory size so memory is always left available for other purposes Associate this pool with users in the IT group # CREATE RESOURCE POOL web_pool MEMORYSIZE '250M' PRIORITY 50 MAXCONCURRENCY 5 PLANNEDCONCURRENCY 1 # CREATE RESOURCE POOL batch_pool MEMORYSIZE '4G MAXMEMORYSIZE '4G' MAXCONCURRENCY 10: 26 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

27 Use Case CEO at Online Retailer Our CEO runs a report every Monday at 9AM. How can we ensure the report always runs on time? 27 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

28 Use Case CEO at Online Retailer Answer PROFILE the query to determine the estimated memory usage Create a resource pool With sufficient MEMORYSIZE Assign the CEO report user to the resource pool using ALTER USER => PROFILE SELECT region, sum(revenue) FROM sales group by region; NOTICE: Statement is being profiled. HINT: select * from v_monitor.execution_engine_profiles where transaction_id= and statement_id=6; NOTICE: Initiator memory estimate for query: [on pool general: KB, minimum: KB] => CREATE RESOURCE POOL ceo_pool MEMORYSIZE '1800M' PLANNEDCONCURRENCY 1 PRIORITY 50; => ALTER USER ceo_user RESOURCE POOL ceo_pool; What are the downsides to this approach? 28 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Monitor Resources System Table RESOURCE_POOLS RESOURCE_REJECTIONS RESOURCE_ACQUISITIONS RESOURCE_POOL_STATUS RESOURCE_QUEUES USERS Description Parameters for all defined pools, including builtin and

30 Monitor Resources System Table RESOURCE_POOLS RESOURCE_REJECTIONS RESOURCE_ACQUISITIONS RESOURCE_POOL_STATUS RESOURCE_QUEUES USERS Description Parameters for all defined pools, including builtin and user-defined pools Statistics on resource rejections and reason for rejection of request Status of resources granted to running queries and queries previously run Current status of resource pools, including memory usage, number of concurrent requests running Current listing of queries waiting in the queue, along with pool and priority information Information on resource pool association to users 30 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

31 QUESTIONS? Please attend our Q&A with HP Big Data experts today Marina Ballroom, Lobby level 10:15 am 10:30 am 12:00 pm 1:00 pm 2:30 pm 3:00 pm 4:30 pm 5:00 pm

Workload Management for an Operational Data Warehouse Oracle Database Jean-Pierre Dijcks Sr. Principal Product Manager Data Warehousing

Workload Management for an Operational Data Warehouse Oracle Database 11.2.0.2 Jean-Pierre Dijcks Sr. Principal Product Manager Data Warehousing Agenda What is a concurrent environment? Planning for workload