Version 1.0. We'll cover these topics:

Transcription

1 Postgres Plus Advanced Server best practices for deployment Version 1.0 Overview This document provides EnterpriseDB customers with best practices and processes for production deployments of Postgres Plus Advanced Server (PPAS). We'll cover these topics: 1. Performance 2. High Availability and Disaster Recovery 3. Connection Pooling and Load Balancing 4. Backup and Recovery 5. Monitoring Please note that the recommendations listed here are not exhaustive, and may change periodically as we review and update our procedures. Further PPAS documentation can be found on our customer portal on our website. Performance Designing and tuning your database system for optimum performance, prior to placing it in production, is a critical aspect of minimizing your downtime and keeping your database user population happy. This section includes general recommendations for: 1. Hardware selection and configuration 2. Database configuration and parameter tuning 3. Tools for troubleshooting performance issues

2 Hardware Selection and Configuration Databases are very bound to your system's I/O (disk) access and memory usage. As such, selection and configuration of disks, RAID, RAM, choice of operating system, as well as competition for these resources, will have a profound effect on how fast your database is. Keep the following in mind when selecting hardware for high performance production boxes: Disks > RAM > CPU Spend most of your budget on high-performance disk arrays; get sufficient RAM and average processors. If you have funds available after your disk purchase, spend it on more RAM. Typical Postgres Plus Advanced Server workloads, like other RDBMS, are fairly I/O intensive, and it's a rare application that taxes the CPU more than the storage sub-system. This applies to small servers as well as high-end boxes. This is not to say that CPUs are not important; however, without a proper storage system, all the CPUs in the world won t help most database deployments. Get More Spindles If there are multiple disks available, most operating systems will parallelize read and write requests on the database. This makes an enormous difference on transaction processing systems as well as applications that do not fit into RAM. Given the minimum disk sizes around these days, you might be tempted to use just one disk, or a single RAID 1 mirrored pair; however, you'll find that using 4, 6, or even 14 disks will yield performance boosts. It should also be mentioned that local (or locally attached) spindles are preferred over remote ones. While iscsi and AoE are better than NFS/CIFS, they still currently lag behind physically connected devices when it comes to raw I/O performance. RAID Strategy RAID is an acronym for Redundant Array (of) Inexpensive Disks. For highly available production systems we recommend RAID 1+0 (RAID 10). RAID 5 is not recommended, because the extra CRC computation and write slows I/O throughput when compared to RAID 10. RAID 6 is even less recommended as the double CRC computation and write is even slower than RAID 5. For a primer on RAID (along with animations showing how data is written to the array), please see Let's discuss RAID 0 (as a base for RAID 1+0), RAID 1, and RAID 1+0(10):

3 RAID 0 The fastest of the RAID levels, RAID 0 is also completely unsuitable for storing any data you value. Technically speaking, RAID 0 isn t RAID, as their is no redundancy. With RAID 0, you have half of your data on one spindle, and the other half on the other. If one of the drives fails, you have lost all of your data. The total amount of space in a RAID 0 environment is the sum of the capacity of both drives. RAID 1 This is the first RAID level suitable for database deployment. Each volume contains complete data and mirrors it with the other. This requires that your RAID controller perform a minimum of two writes for each single write operation. Your read performance may not suffer too much, as most controllers will either balance the load of the read against different spindles, or will read the RAID 1 volume similar to a RAID 0, reading 50% from each spindle in the mirror. The usable amount of space in a RAID 1 is the capacity of a single drive. RAID 10 (RAID 1+0) Disk Controllers A nested level, RAID 10 (or 1+0), is the preferred configuration for Postgres Plus Advanced Server. It combines the redundancy benefits of RAID 1 with the performance enhancements of RAID 0. The usable amount of space is half the total capacity of the drives. When selecting a controller, look for benchmarks that demonstrate high performance on both sequential and random reads and writes. We've had good success with the HP P410i and higher models, and LSIs (though not the LSI re-brands.)

4 Use a Battery Backup for Write Cache Your storage device usually buffers data in a cache before writing it to disk in order to improve disk I/O performance. Get a write cache with a battery backup (BBU), so that a power failure will not result in loss of data and hence data integrity. Most BBUs have about one week of standby time. As long as you bring the machine up within that period, you will not lose data that was pending before the system crash or power failure. Your cache controller should run in Write-Backed Mode when the battery is charged and automatically degrade to Write-Through Mode when the battery fails and is in need of replacement. Cheaper controllers will not do this degrade automatically and will continue to run in Write-Backed Mode even when the battery is dead. This could lead to data loss in the event of a power issue. Check your controller s specifications carefully! Deploy the database on separate disks and LUNs We recommend that a production deployment of PPAS have at least three separate drives (or sets of drives): one for the transaction log, one for the data files, and a third for indexes. In some cases, depending on workload, database size, and available storage, the data files and indexes can be sub-divided onto additional storage areas. This is done by creating TABLESPACEs and mapping them to the mount points of a particular drive; find more details about this in the Postgres Plus Advanced Server documentation. In certain scenarios, you might even mount pg_stat_tmp to a ramdisk or tmpfs. If you can only accommodate two separate drives, use one for the transaction log and the second for the data files and indexes. Understand your choice of filesystem Equally as important as using multiple LUNs, disks, and mount points is your choice of filesystem. Understand the semantics and performance tradeoffs of a given filesystem before deciding to use it on a given mount point and you will save yourself headache down the road. For example, there is no need to use a journaling filesystem for the transaction log mountpoint due to the way the transaction logs are written whole pages at a time using fsync (and the fact that pages are checksummed). As such, using ext2 for the pg_xlog directory is perfectly fine. Additionally, due to a bug in the way ext3 handles fsync, we do not recommend its use for any of your database mounts. Instead, look at ext4 or other journaling filesystems. Be extremely wary of filesystem deduplication! It can easily deduplicate you into data loss. Once a filesystem (or filesystems) has been chosen, you should use the pg_test_fsync binary that ships with PPAS to understand the fsync performance characteristics of your system. While PPAS

5 defaults are typically the correct choice, there is merit in verifying the choice against your specific configuration. SSD While it is still relatively early to paint the SSD market in broad recommendation brushes, there are a few things to make particular note of. Be extremely wary of any SSD that performs hardware deduplication at the block level (sometimes labelled as transparent deduplication ). Deduplication at the filesystem level is risky enough, but deduplicating in the hardware at the block level when the filesystem is unaware of it is simply asking for data loss. Most current filesystems depend on duplicating metadata across the filesystem for their own housekeeping and consistency checks. Having the hardware secretly deduplicate this information can be disastrous! Additionally, a growing number of SSD manufacturers now offer built-in cache controllers but unfortunately these controllers are either not battery backed at all or are hard-wired in Write-Backed Mode and cannot degrade to Write-Through Mode when the battery is discharged. In the event of a power event, you will lose whatever was in the cache on these devices. Hopefully this situation will improve as more SSD devices become enterprise class. RAM Generally speaking, you cannot have too much RAM for production workloads. Most of our high-end production customers work with 32 GB or more of RAM for their mission-critical deployments. Processor There are two main points you need to consider while choosing CPUs for Postgres Plus Advanced Server: processor architecture (32-bit vs 64-bit) and the number of concurrent connections your workload requires. A 32-bit CPU restricts the size of shared memory regions, forcing PPAS to run with a smaller-than-desirable cache. This can have a severe adverse impact on performance. Modern CPUs are 64-bit, so this is unlikely to be an issue. Postgres Plus Advanced Server is process-based, so having more than one CPU helps you to scale highly concurrent applications. For workloads with very high connection counts, having additional CPUs can be helpful, especially for read-heavy workloads that do a significant amount of calculations on the database server. At this time we don t have more specific CPU recommendations; look for them in a future version of this document.

6 Database Configuration and Parameter Tuning Properly configuring and tuning your database to match your production workload can result in orders of magnitude in performance benefits. There are two places to find EDB best practices recommendations for database tuning: 1. Postgres Plus Advanced Server Guide, section , Top Performance Related Parameters: This document, available on EnterpriseDB s website, gives detailed descriptions and recommended settings for database configuration parameters. 2. Postgres Expert. Postgres Expert is part of Postgres Enterprise Manager (PEM), included with any PPAS subscription. It evaluates your current hardware and database settings and recommends changes to your database configuration parameters to maximize performance. PEM is covered in more detail elsewhere in this document. Tools for troubleshooting performance issues Postgres Plus Advanced Server has a number of tools to help you troubleshoot performance problems. They are all documented with detailed examples in the Postgres Plus Performance and Scalability Guide. Specific tools our field engineers and customers find especially helpful are: The Index Advisor Dynamic Runtime Instrumentation Tools Architecture (DRITA) DBMS_PROFILER Package High Availability and Disaster Recovery Clustering for Availability Computer systems can and do fail. For PPAS systems that have extremely high uptime requirements, we recommend a high availability (HA) configuration using a shared storage array and Red Hat Cluster Suite (RHCS). This industry-proven clusterware software monitors the Postgres Plus Advanced Server service for failure and can restart the failed server on a second node. This failover can happen in seconds, which means your end users will be able to continue their work with minimal impact. There are many other solutions that can be used to handle the

7 failure of the primary database server; however, EnterpriseDB has the most experience with this particular solution and we recommend it as a proven HA configuration. Detailed deployment instructions are beyond the scope of this document; so we ll go over a high-level description of how it works and some of the options that go with it. The diagram below illustrates this active/passive architecture with the following components: two Postgres Plus Advanced Servers, one of which is configured as the active server shared storage connectivity between the servers connectivity between each server and shared storage, so each server can mount any of the filesystems on the shared storage clustering software The cluster software monitors the health of the cluster and takes action when a failure is detected. The cluster software sends a heartbeat signal between the servers. The successful receipt and acknowledgement of this heartbeat indicates that the cluster is healthy and no action is needed. If the heartbeat signal is lost, that means something has happened and a failover is initiated, following these general steps: 1. The clusterware detects the failure 2. The clusterware fences off the failed node 3. The clusterware mounts the shared disk to the standby node 4. Postgres Plus Advanced Server is started on the standby node 5. The former standby node becomes the active node. Since this Active/Passive solution uses shared storage, there is no

8 data loss. All of the data in the database will be accessible to the passive node once the clusterware mounts the storage on the passive node. You can configure the clusterware to be as aggressive as you wish in detecting a failure. You also have the ability to control the length of time it will take Postgres Plus Advanced Server to restart on the passive server. This startup time will be dependent on the amount of transactions that need to be replayed to get the database into a consistent state before making itself available to new operations. This replay, or roll forward, period is directly related to how often you perform checkpoints in the database. In order to ensure a fast startup, configure your checkpoint mechanism to checkpoint often. In read-heavy environments (i.e., those with few changes), frequent checkpoints will not cause any substantial overhead. By adjusting the settings of the clusterware and Postgres Plus Advanced Server, you can achieve failover times well within your requirements. EnterpriseDB has one trading company running their main trading system in an Active/Passive cluster and they failover in 8 seconds upon detection of a failure. Disaster Recovery The active/passive clustering just described provides high availability, but will not protect you if something happens to your data center. Replication provides a solution to protect you from such failures. In PPAS there are several options available to replicate your data to a mirror database. We ll discuss two of those options: log shipping and streaming replication. Depending on which option you choose, the mirror database can be available for querying. Log shipping Log shipping is a good way to implement DR in case you must use a version of Postgres earlier than 9.0. Log shipping involves sending your primary cluster s Write-Ahead Logs (WALs) to your standby server. The standby cluster is always in recovery mode and will not accept connections. Every time there is a WAL log file switch on the master cluster, the archive command will be triggered and will copy the new WAL log file to the slave cluster. The recovery process in the slave cluster then applies the new archive to the database. With this configuration, replication is cluster-wide; it is not possible to replicate only one database or a set of tables in a database. Streaming Replication Asynchronous Streaming Replication was introduced in PPAS 9.0. As of PPAS version 9.1, you also have the option to use synchronous replication, though typically we don t recommend it for remote

9 disaster recovery because of the associated performance hit. However, in a disaster recovery situation, you could experience data loss on your secondary database when using asynchronous replication. You need to compare the data loss potential of asynchronous replication to the performance penalties of synchronous replication and decide which is best for you. Connection Pooling & Load Balancing Connection poolers reuse existing database connections, reducing the overhead required to create new ones. Load balancers distribute queries across databases in your replication pool. pgpool-ii and PgBouncer both provide connection pooling; pgpool-ii provides load balancing as well. pgpool-ii pgpool-ii is a middleware solution that would reside between a Postgres Plus Advanced Server and an application server. For more information about pgpool-ii, see the project page: PgBouncer PgBouncer is a pre-bundled enterprise module installed by default with Postgres Plus Advanced Server. It is a very lightweight connection pooler and is the preferred choice for customers who need

10 only connection pooling. For more information about PgBouncer, see the pgbouncer wiki: General Recommendations on Connection Limits and Pooling Coming up with a generic recommendation for connection limits is difficult, because it can vary so much depending on the activity on each connection. We have customer deployments with max_connectionsset to over 2500, but with the majority of connections remaining idle. Other factors to consider when deciding how to limit the number of connections include the amount of data transferred on each connection, and if the transactions are primarily read or write. With those factors in mind, here are some general guidelines: 1. On a server with 64 GB of RAM and 8 cores, a value of max_connectionsset to 500 or below works well for most web-facing workloads. 2. Always consider what work_memis set to when adjusting max_connections. A high value for both can result in system swapping. Since work_memgrows dynamically depending on the size of a SORT operation, it is difficult to determine in advance if this will be an issue. Monitor it closely and adjust accordingly. 3. If you find yourself needing a bigger value for max_connections, consider using a connection pooler. 4. In general, process creation is more expensive on Solaris than Linux. Therefore, connection pooling on Solaris will yield greater returns on performance. 5. If you have a number of application servers between a load balancer and a PPAS cluster, place the connection pooler on the same host as each application server. This design, as opposed to placing a single connection pooler on the database, has been an area of much debate internal and external to EDB. After observing what has worked for our customers, this is the recommendation we have settled on. Specific details behind this recommendation are beyond the scope of this document. 6. As of this writing, we recommend PgBouncer over pgpool-ii. Backup and Recovery Create, test, and document a backup and recovery policy prior to implementing your Postgres Plus Advanced Server production environment. Your plan should cover: 1. How frequently will backups be made 2. How long will backups be retained 3. How frequently will backups be tested

11 4. What is the test plan for a backup 5. Where will backups be stored (on site / off site, on line / off line media). 6. Target completion time 7. Target recovery time PPAS supports three types of backups: SQL dump, filesystem level backup (usually offline), and continuous archiving (online backups). Details about these backup methodologies can be found here: General backup tips: Take online backups using pg_start_backup() and pg_stop_backup(). Its prerequisites enable you to take advantage of some other key production features such as Streaming Replication, Point in Time Recovery (PITR), and Hot Standby. In addition, this offers the least disruption to the system. PPAS ships with an executable named pg_basebackup that automates much of this process. Automate your backups. Configure alerts for successful and failed backups: automate scripts to send to a monitored address or alert your existing monitoring software. Create a data retention policy. Have scheduled backup verification at least once a quarter, if not more frequently. Maintenance Monitoring Postgres Enterprise Manager from EnterpriseDB is an outstanding tool for monitoring Postgres Plus Advanced Server. It's available as part of our regular database subscription services, or as a standalone subscription. We strongly recommend you provision an additional small server with your database deployment to host the PEM database. Configure Alerts Alerts are generally application- and system-dependent. Consider the types of events you are concerned about based on the characteristics of your system, and set up PEM alerts for them. At a minimum, set up alerts for CPU usage, server down, swapping, and low disk space. Check out these example alerts to get some ideas: High usage of swap memory

12 CPU Load over 50 percent Number of active connections exceeds 85 It's been over 24 hours since the HR database was analyzed There is over 100 megabytes of bloat in a table The HR database has exceeded 200 Gig. in size There is only 100 MB of disk available on the host server A query is doing sequential scans on the Employee table A table under heavy update hasn't been vacuumed for 3 days In addition to those available through PEM, you should have alerts for the following : Network disconnections between the database server and a remote server where archives are copied or applied Successful completion or failure of automated backup scripts Availability of the pooler software (pgpool-ii or PgBouncer), if you are using connection pooling Availability of streaming replication or standby Configure alerts to go to an address that is monitored by individuals and/or to an SNMP trap handler to be integrated into your existing fault management system. Develop a plan for when an alert does enter the alerted state: what actions will be taken and who is responsible for taking those actions. Having advance notice of problems allows you to make corrections before they cause production issues, and is the best way to maximize your uptime. Review your Dashboards Daily PEM has an outstanding quick look -style dashboard that provides a weekly snapshot view of the health of your databases, database servers, and hosts. You can easily see problem areas such as swapping, connection spikes, and rapidly shrinking available disk space. Reviewing the dashboard on a regular basis enables you to identify and correct problems long before they result in a bad user experience or even an outage. Note: You can change PEM's weekly default display to a longer or shorter time period to suit your own needs. Capacity Manager One of PEM's strongest features is the Capacity Manager. It allows you to collect statistics, view trends, and do predictive analysis based on historic data. There are hundreds of statistics you can monitor and create canned reports for. We recommend you decide which reports you need prior to your production deployment. Generally, you should have reports on the same statistics for which you have alerts. User Accounts and Auditing

13 User Accounts Create separate accounts and privilege levels for database administration and for application usage. (A good general rule is don t give applications global administration privileges.) Do not allow multiple users to share a single account; each user should have her own account. The ability to associate each user with a single set of credentials provides accountability for innocent mistakes as well as malicious acts. When users know there is visibility into their actions, they are encouraged to be careful, and discouraged from doing something intentionally to harm your business. Log and Audit For most deployments, we recommend logging and auditing DML and DDL changes as well as ERRORs, and to associate these changes with specific users, as these are significant events in a production environment. DDL Updates During Operation Making changes to your production DDL can adversely impact database performance. Here are some general guidelines to reduce that impact: Tables can be safely added without creating concurrency issues for the database deployment. When adding a column to a table, the best approach is to add the column with a default value of NULL in one transaction, and then add the values in a separate second transaction. Use the CONCURRENTLY key word when adding indexes. This will take longer, but won t acquire any exclusive locks on tables and will minimize the disruption on a running database.