Heimdall Data Access Platform Installation and Setup Guide

Transcription

1 Heimdall Data Access Platform Installation and Setup Guide

2 Heimdall Data Access Platform Installation and Setup Guide Heimdall Data Access Platform Installation and Setup Guide

3 1. General Information Overview of the Heimdall Data Access Platform (HDAP) Installation Overview System Requirements Windows Installation Non-Windows Install Changing default port Using the Sample Traffic Generator Starting the traffic generator Application Installation General Installation Cold Fusion Configuration Performance Analysis Baseline Performance Measurement Benchmarking Heimdall Support Contacts iii

4 iv

5 Chapter 1. General Information 1.1. Overview of the Heimdall Data Access Platform (HDAP) HDAP is composed of two pieces-the HDAP Driver and the HDAP Central Server. The driver can be classified as either a type 3 (middleware) or type 4 (native Java) driver, depending on how you view the definition. Unlike most type 3 drivers, queries are not implicitly translated by the driver or passed to another device. Instead, the Heimdall Driver leverages existing vendor JDBC drivers executing on the client system to access the database, adding very little latency. All features implemented by the HDAP execute on the same system as the application, providing scalability beyond what other middleware solutions provide. The central server component operates as a control plane for one or more application servers leveraging the driver, and provides configuration management, logging, and failover logic for the driver component. A brief list of the major supported features is below: Vendor and protocol neutral support for all features Provides caching both in-jvm and via external Grid cache interfaces (Memcached, Redis, EHcache, Hazelcast, and others) Regular expression and table aware policy configuration for caching, providing easy yet reliable cache behavior High availability failover for databases in simple or complex topologies including active, standby, and read-only database servers Read/write split of traffic to read-only database servers Policy-based redirection of queries to alternate data sources based on regular expressions for optimization and/or sharding Connection pooling for applications that don't include connection pooling, or have inefficient connection pooling logic Query transformation for quickly resolving poorly written queries Provides run-time reconfiguration of data sources for maintenance activities Log SQL queries & JDBC methods, selectively or globally, for performance analysis or postmortem problem analysis Query analysis for "one-click" caching based on the best cache candidates Access control for users, including subnet and host based access 1

6 2

7 Chapter 2. Installation 2.1. Overview The free evaluation version of HDAP is a fully functioning system with the following constraints: Only 8 cores of processing power can be managed by the server at a given time 30 day license from the time of install If a less constrained license is needed to properly evaluate the HDAP, please contact Heimdall support at support@heimdalldata.com to request a license that meets your requirements System Requirements The HDAP requires a JRE of 1.6 or higher and requires at least 1GB-2GB of free memory for the server component, depending on system load. The driver component has a configurable RAM cache, and will need about 200MB of additional heap space in addition to the configured cache size. It is also highly recommended that anti-virus tools be disabled during install, as in particular when installing a private JRE, some libraries are blocked from being installed by them Windows Installation HDAP can be downloaded, installed, tested, and configured for your environment in less than 5 minutes. Here are the easy steps: 1. Downloading - First, download the HDAP (Windows 64 bit) installer package to your computer from 2. Installation - The HDAP installation package contains the driver, the central server, an optional private JRE, grid API support libraries, and a sample traffic generator. A default install will include the following options: 3

8 Chapter 2. Installation Custom Install Selected With Feature Tree Expanded, and Default Options Each feature is described below: A. Heimdall Server The central server component necessary to manage the Heimdall driver component. This only needs to be installed on a single machine, and can be used from many machines (license permitting). B. Heimdall Service If you wish to install the Heimdall Server as a system service under windows (recommended), please leave this selected. On install, the service will be started automatically. C. Private JRE Only necessary to have selected if no system-wide JRE is available. This should automatically be selected for installation if this condition applies. D. Example Configuration If installed, the Heimdall Server component will startup with a configuration already in place. For evaluation purposes, this is recommended. This configuration can be modified post-install. E. Driver Library This can be installed with the Example Configuration or separately, but is recommended, as it simplifies the configuration of database access through Heimdall. The driver library can be added to or modified after installation. F. Heimdall Driver 4

9 Windows Installation This component only needs to be installed on a system that an application will be run. The Heimdall Driver install location can be modified to point to the library directory of an application directly to simplify the install process. G. Grid API Libraries When installed, a lib directory will be created under the install location of the Heimdall Driver, with supported grid cache API libraries. Usually, if used in an application, this is not necessary, as they will already be present in the classpath, and will be used automatically by Heimdall. Generally, this is used for testing the demo application with grid interfaces. H. Traffic Generator Example To provide a simple set of traffic for demonstrating or understanding features, this feature installs into the traffic directory under the Heimdall Server install directory by default. 3. Service Startup - If the Heimdall Service component was selected, the Heimdall service should have been started automatically, and the console should have been opened in the default web browser. You can start configuration immediately. To start and stop the service, use the Services manager: 4. Manual Startup - If the service installation is not used, the server component can be started manually by navigating to the install location (default c:\program Files\Heimdall) and executing the script "serverstart.bat" as administrator: 5

10 Chapter 2. Installation 5. Accessing the Central Manager - On install, if the service component was installed, you should automatically be directed to the console at This same URL can be used to open the console at any time. 6. Getting Configuration Help - Each tab has a "help" option at the upper right corner of the tab that provides access to online documentation. Please refer to this documentation for detailed. As an example, on the Virtual Database tab, you would use the highlighted button as shown below: 2.4. Non-Windows Install 1. Downloading - First, download the OS independent install package at download.heimdalldata.com/downloads/heimdalldemo.tgz. The only host software requirement is that the system supports a JRE 1.6 or higher. 6

11 Changing default port 2. Installation - The HDAP installation package contains the driver, the central server, grid API support libraries, and a sample traffic generator. Decompress the archive into a directory (/opt is suggested). 3. Service startup - As different operating systems have different requirements for startup, a simple script is provided that can be called from either an init script, from rc.local, or by another startup mechanism. The script is "heimdallserver.sh" and will accept the options for "start" or "stop" in order to start or stop the Heimdall server Changing default port The default port used by the Heimdall Server is tcp port In order to change the port, simply create a file named "application.properties" in the install directory, and add a line like: server.port = X Where X is the desired port number. 7

12 8

13 Chapter 3. Using the Sample Traffic Generator 3.1. Starting the traffic generator The included traffic generator source is installed by default into the directory c:\program Files\Heimdall\traffic, and can be run by executing the runme.bat also in the same directory. On the initial run, the source will need to be compiled by a locally installed javac, which is included in the Java JDK environments by default. The private JRE that may be installed to support the service component does not include a javac executable. If a JDK is needed, then it can be downloaded from Oracle. Once javac is available, the runme.bat script can be executed as administrator to compile the source, and then to execute the result (a warning will be presented if this does not work): Upon execution, the traffic generator will use the sample configuration that is by default installed into the Heimdall Server, and will leverage a database available on the public Internet to allow demonstration of the various features of the HDAP. Once started, the server console should provide a view such as below on the dashboard, which can be used to confirm that traffic is being passed properly: 9

14 Chapter 3. Using the Sample Traffic Generator From here, feel free to explore the features that the HDAP provides via the online help, and through the sample configuration that is provided. 10

15 Chapter 4. Application Installation 4.1. General Installation When starting the process of installing Heimdall into an application, in general, the following steps should be taken: Select an application server/client that can be restarted, and traffic driven through outside of production. Insure that the system selected is operating normally-this is extremely important, as often issues arise during Heimdall install that are not related to Heimdall at all. Copy the Heimdall driver jar file into the classpath of the application. Normally, it is best if you find the current JDBC connector jar file, then copy the heimdalldriver.jar into the same directory. Find where the application configures the JDBC class path and Java class for the the JDBC connector. For example, with the Atlassian Bitbucket server product, you will find the configuration in: /var/atlassian/application-data/bitbucket/shared/bitbucket.properties: #>******************************************************* #> Migrated to database at jdbc:mysql://localhost:3306/bitbucket?characterencoding=utf8&useunicode=true&sessionvariables=storage_engine %3DInnoDB #> Updated on T20:33:48.292Z #>******************************************************* jdbc.driver=com.mysql.jdbc.driver jdbc.url=jdbc:mysql://localhost:3306/bitbucket?characterencoding=utf8&useunicode=true&sessionvariables=storage_engine %3DInnoDB jdbc.user=user jdbc.password=password As another example, for Atlassian Jira, the configuration is in: /var/atlassian/application-data/jira/dbconfig.xml:... <url>jdbc:mysql://localhost:3306/jira?useunicode=true&characterencoding=utf8&sessionvariables=storage_engine=inn url> <driver-class>com.mysql.jdbc.driver</driver-class> <username>root</username> 11

16 Chapter 4. Application Installation <password>a1k81ucutr</password>... Create a data source object with the configured JDBC driver, JDBC URL, user and password. Verify that the central server can connect to this data source object. Create the application's VDB. Edit the application configuration to use the JDBC URL and Class specified in the VDB configuration. If you are using authentication on your Heimdall Server, use the username and password to authenticate to Heimdall as the JDBC username and password, as the database username and password will be configured via the data source. Note: If you have dynamic or multiple users and passwords, and the application controls this, you can use a JDBC query parameter of "hduser" and "hdpassword" to provide Heimdall Server credentials. Start the application. Check the application console logs for messages similar to the following, which indicates a proper startup of the Heimdall Driver. [ :06:02.344] Heimdall: Retrieving updated configuration for URL: [ :06:02.345] Heimdall: getting config from: [ :06:02.545] Heimdall: heimdall: Startup performed: com.heimdalldata.log.eventlogger@7de87192 [ :06:02.546] Heimdall: Initializing logger: com.heimdalldata.log.eventlogger@7de All messages being generated by the Heimdall Driver will start with the format [YYYY-MM-DD HH:mm:ss.SSS] Heimall: message. If they can't be found in the application logs, it is likely that the driver was not initialized by the application properly, and other application errors should be checked. Common issues with configuration: 1. The driver was not found in the application class path 2. The JDBC URL or Classname was not configured 3. Port 8087 is not open for access to the Heimdall Server 4.2. Cold Fusion Configuration To install the Heimdall Driver into Cold Fusion, please follow the following steps. 1. Copy the driver into to the Cold Fusion lib directory (by default C:\ColdFusion11\cfusion\wwwroot\WEB-INF\lib). 2. In the Cold Fusion GUI, navigate to the Data & Services > Data Sources window: 12

17 Cold Fusion Configuration 3. Using the "other" data source type, add a new data source, and configure it with the appropriate Heimdall JDBC URL, but with all other settings identical to the original data source used previously: 13

18 Chapter 4. Application Installation 14

19 Chapter 5. Performance Analysis 5.1. Baseline Performance Measurement If you can't measure it, you can't improve it - Peter Drucker In order to determine how effectively Heimdall is able to improve performance, it is necessary to first create a reproducible performance test that can be executed many times to get statistically meaningful performance measurements. There are many different approaches to doing this, and tools to provide this. Some of these include: For web applications, "true" user performance can be measured with Dynatrace's or Catchpoint's products. For local Web based application performance benchmarking, Jmeter or LoadRunner can be used. For GUI applications that rely on databases, a tool such as Cucumber ( can be used. For any such testing, it is important to insure that the results of the baseline are reproducible, and consistent between test runs. Once a baseline performance test has been run, Heimdall can be installed, but several considerations need to be made first: Restart the application between each test. As Java can optimize execution over time, and an application can also leak memory or cache on it's own, it is important to make sure the application is restarted to get consistent results between test runs. Install the Heimdall Server component on a separate server for performance testing, as the additional load from log processing and memory overhead may slow down the application Benchmarking Heimdall Once a reproducible baseline test has been created, the Heimdall Driver can be tested: Test the application with the VDB "pass-through" option first, to ensure that the installation behaves with a similar performance profile. Some applications, if you install a custom driver, will behave differently vs. a built-in driver. Cold-Fusion for example, issues different JDBC calls for the MySQL driver vs. a custom driver. The Heimdall driver has been optimized to detect this behavior, but other applications may also behave differently with different drivers. Assuming similar performance has been observed in pass-through mode, test without pass-through enabled, but no other logging or policies in place. Again, performance should be close to identical vs. the non-heimdall performance profile. Next, test with SQL logging enabled. It is possible that this will induce a small (no more than 5-10%) reduction in database performance. With SQL data gathered, utilize the Analytics system to configure caching for appropriate queries. Keep in mind that not all recommendations make sense, but the recommendations can narrow down what policies to consider. Test with caching enabled, but disable logging again. This should be the normal behavior in production, although sampling can be used to provide an overview of traffic without significant performance impact. 15

20 Chapter 5. Performance Analysis In the event that performance is not improved, or is negatively impacted when it shouldn't be, enable method and result set method logging on the VDB, and test with a very low load test, and then provide the logs from the logs directory (default c:\program Files\Heimdall\logs) to Heimdall Support for analysis. Hints for successfully benchmarking without test artifacts: Monitor the system performing the test for CPU load and memory saturation. Often, a benchmark tool is actually measuring it's own ability to generate traffic, not the system under test. Try to avoid using virtualized environments for testing. Using dedicated hardware for clients, servers and networks will help reduce the number of variables that can impact a test. Keep an open mind on results, in particular when results look particularly good or bad, as either may indicate an issue in test methodology. Always consider variables relating to the test environment in mind. If large numbers of connections are being established and torn down, consider TCP port use and reuse. Searching on the web for "windows ephemeral port exhaustion" will bring several topics about this common issue during benchmarking or heavy server load. Ask for help in analyzing results and the overall test setup! Heimdall Support can work with you to ensure that the appropriate setup is used for reliable results. 16

21 Chapter 6. Support 6.1. Contacts For any issues, comments or suggestions, please use any of the following contacts: Phone: HDDATA4 ( ) Website: 17

22 18