DSE Setup Guide for Oracle Databases

Transcription

1 DSE Setup Guide for Oracle Databases I. State Steps (Pre-DSE Installation and Setup): States will have to complete the following steps before installing and setting up DSE. 1. Create a new schema in the SDWIS State database named DSE_MATRIX. The user or role permissions granted to the schema should be CRUD. The new schema does not touch existing SDWIS/State schema. A. DEFAULT TABLESPACE [PRIME_DATA] (State will replace PRIME_DATA with their table space name). CREATE USER DSE_MATRIX IDENTIFIED BY DSE_MATRIX. As a part of DSE setup, Attain s code will create the required batch tables in the DSE_MATRIX Schema. These batch tables will keep track of information for logs and other reporting purposes. II. DSE Installation and Setup: Verify if JAVA JDK version 1.7 or higher is installed. If not, JAVA JDK has to be downloaded and installed. Please click on the following link to install the latest version of JAVA JDK: Note: JDK versions below 1.7 are not supported. JRE (Java Runtime Environment) is not sufficient for DSE installation. If you choose to have multiple versions of JDK installed on your system, please make sure you point to the correct Java.exe file before starting the DSE Server. Instructions on how to point to the correct Java.exe file are mentioned below. - Below is an example showing multiple versions of JDK Note: DSE installation would support Tomcat version 6 and above. If you already have tomcat v 6 installed (eg. For SDWIS / STATE), you can run the DSE server in the same instance. In some instances, you will have to increase the memory size to run DSE. If you choose to use an existing instance of Tomcat, please skip to Step 3. Step 1: Create the following folder structure In the C drive: A) Create a new folder called dse. (Note: Folder name should be in lower case)

2 B) Inside the dse folder, create a new folder called dev Step 2: Downloading Tomcat A) Download tomcat (apache-tomcat zip) from the CMDP Help Desk link ( B) Unzip and place the apache-tomcat zip file in the dev folder created in Step 1. Note: If you wish to use tomcat v6, it can be downloaded from the CMDP Help Desk using the following link ( file name: apache-tomcat src.zip Step 3: Coping DSE server Files to the Final folder A) Download the following files from CMDP Help Desk link ( and save to the specified location: File Name Location Example dse_config.properties User defined folder for DSE files ($DSE_HOME). C:\public\data\$PROJECT C:\DSE_HOME setenv.bat Base folder where the running configuration of Tomcat exists C:\dse\dev\apache-tomcat \bin ($CATALINA_HOME) Setenv.bat_as_env_vars User defined folder for DSE files ($DSE_HOME). C:\public\data\$PROJECT C:\DSE_HOME CypherTool.jar Refer to Instructions to use Cypher Tool.docx document for instruction to use Cypher tool. C:\public\data\$PROJECT C:\DSE_HOME dsewebadmin.war Tomcat -> Webapps folder C:\dse\dev\apache-tomcar \webapps ojdbc jar Tomcat -> Lib folder C: \dse\dev\apache-tomcat \lib Step 4: Installing SSL Certificates that allow DSE to access Web Services. A. Download the following Java security files from CMDP HELP Desk ( 1. local_policy.jar and US_export_policy.jar 2. Place (Replace if necessary) the jar files in the following location C:\Program Files\Java\jdk1.7.0_25\jre\lib\security Note: C:\Program Files\Java\jdk1.7.0_25 is the JAVA_HOME location used for DSE. ** IMP NOTE: Above mentioned.jar files work for JDK version 1.7 and above. For example, if you do not have jdk1.7.0_25 but you have jdk1.7.0_75, you may use jdk1.7.0_75 instead. B. Download the following Certificates from CMDP HELP Desk. ( ) 1. For Pre-Production download certificate titled Pre_ProdEpa.gov.crt. For Production download certificate titled ProdEpa.gov.crt. 2. Save the certificate files in { JAVA_HOME}\bin folder. 3. The following command has to be executed from the folder where the certificates are saved.

3 a. Open the command prompt (In administrator mode) and navigate to the folder where the certificate is saved. b. Run the following command for Pre-Production. keytool.exe -import -alias Pre_ProdEpa_gov -file Pre_ProdEpa.gov.crt -keystore "{JAVA_HOME}\jre\lib\security\cacerts" c. Run the following command for Production. keytool.exe -import -alias ProdEpa_gov -file ProdEpa.gov.crt -keystore "{JAVA_HOME}\jre\lib\security\cacerts" 4. When prompted for a password, enter: changeit Note: Replace JAVA_HOME with the location JAVA_HOME is pointing to. Step 5: Executing the installation file to start the server. This step is for users running tomcat from the command prompt, otherwise go to Step 6. A. Open command prompt by pressing Windows button + R at the same time B. Type in cmd and click OK C. Once the command prompt opens up navigate to the apache-tomcat by typing in the following commands: 1. cd C:\ 2. cd dse\dev\apache-tomcat \bin 3. setenv 4. catalina run The command catalina run will start and establish a connection between CMDP and the primacy agency s database Step 6: This step is for users running Tomcat as a service. Step 7: A) Start Tomcat from the windows console. A. Open up a web browser and enter the below link in the address bar to make sure the server is up. a. b. Click on Jobs to view jobs started by DSE. c. Click on s2c_job_parallel to view jobs that sync data from State DB to CMDP. Click on cmdp2statejob to view jobs that place XML Sample result files in the state XML folder. d. The latest ID should have a COMPLETED status. NOTE: Instruction to customize tomcat port can be found in section III of this document. III. Accessing tomcat through custom port number: The DSE can be installed in a different instance of tomcat and can be accessed through a custom port. Follow the steps below to customize a tomcat port number. (Make sure your tomcat instance is not running before changing the port number).

4 Step 1. Under tomcat folder, open on conf folder Step 2. In conf folder, open server.xml file in notepad++ or a similar application. Step 3. In the server.xml file, search for 8080 (default port number) by pressing ctrl + f keys.

5 Step 4. Change 8080 to a custom port number. (eg. 8181) Step 5. Start tomcat, wait for few mins and open tomcat manager through a custom port. (eg: localhost:8181/dsewebadmin) Step 6. Click on Jobs to access jobs being run under DSE application. IV. Accessing Error Log in tomcat Error logs can be accessed in your tomcat localhost link. Below are the instructions to access your Error Log. Note: Tomcat admin credentials might be required to access the tomcat localhost link. - Make sure tomcat is up and the DSE is running. - Enter localhost:(your tomcat port number for DSE)/dseWebAdmin/jobs in your browser. o Eg: localhost:8181/dsewebadmin/jobs - Click on s2c_job_parallel to view jobs related to Inventory and Laboratory Profile updates. - Click on cmdp2statejob to view jobs related to samples migration to the XML Sampling Application. - Click on the Last JobExecution status to view if any error were generated.

6 o Status could be FAILED or COMPLETED - To view more details on the error, users can open the dse_app.log files under the tomcat directory -> logs folder. o DSE generates multiple log files based on how much data is logged in the log files. If the dse_app.log file reaches 10MB, DSE will push the data from dse_app.log into dse_app.log1 and start to write new data into dse_app.log file. DSE has been programmed to generate up to 40 log files before overwriting existing log files. o A user can search for "ERROR" or IllegalStateException in dse_app.log file to navigate to Error description. o Please refer to examples of different kinds of errors in the Appendix. V. CRON Job Customization: CRON frequency can be customized according to individual primacy agency needs by changing the CRON parameters in the setenv.bat file or in Java Options section under tomcat properties window -> Java tab. The CRON consist of six parts and the parts are separated by a space. For example: if you want to run a CRON every day at 12 AM you will define it as follows: set scheduling.job.cron.s2c= * * MON-FRI The above configuration is for tomcat that runs from the common line, but for tomcat that is running as services the values will be as follows: -Dscheduling.job.cron.s2c= * * MON-FRI CRON expression is represented by six fields: Second(XX), minute(xx), hour(xx;24 hour format), day of month(xx), month(xx or L), day(s) of week (DDD) (*) means match any */X means "every X" L last day of the period (e.g. Month)? ("no specific value") - useful when you need to specify something in one of the two fields in which the character is allowed, but not the other. For example, if I want my trigger to fire on a particular day of the month (say, the 10th), but don't care what day of the week that happens to be, I would put "10" in the day-ofmonth field, and "?" in the day-of-week field. Recommended setting: For initial load set the CRON job to run once a week (E.g. set scheduling.job.cron.s2c= * 01 MON 2017). Following the initial load, change the CRON job to run once a day. Expression Meaning ========== ======= * *? Fire at 12am (midnight) every day ? * MON-FRI Fire at 10:15am every Monday, Tuesday, Wednesday, Thursday and Friday *? Fire at 10:15am on the 15th day of every month L *? Fire at 10:15am on the last day of every month * * MON-FRI scheduled to run 15 minutes past each hour but only during the 9-to-5 "business hours" on weekdays Please pay attention to the effects of '?' and '*' in the day-of-week and day-of-month fields.

7 VI. DSE logic to populate last sync date in the delta queries DSE will follow the below logic to populate last sync date while migrating delta records. a) If "dse.s2c.first.pull.date" value is added to environment variables, DSE will choose this date as the last sync date. b) If "dse.s2c.first.pull.date" environment variable is missing, DSE will query for END Date of the last successfully COMPLETED S2C job in BATCH_JOB_EXECUTION table and use it as the last sync date. c) If "dse.s2c.first.pull.date" environment variable is missing, and there was no last successfully COMPLETED S2C job in BATCH_JOB_EXECUTION table, DSE will use the current run date as the last sync date.

8 Appendix: Validation Errors: 1) IllegalStateException: watersystemid:user Not Authorized for Water System Id. The most common reason this validation error occurs is because the first two digits of PWS ID do not match the primacy agency that the state admin role is assigned to. This could be a state data quality issue which can be corrected in the primacy agency s database. 2) IllegalStateExceptionwaterSystemId:Water System Id must be 9 characters. The first 2 characters represent the state code and the next 7 are the Water System Id. This validation error is generated when the PWS ID does not meet the 9-characters requirement. This is another state data quality issue and can be corrected in the primacy agency s database. 3) validationerrors={watersystemid=invalid Water System Id.} This validation error is generated while processing the Facility step or Sampling Point step. If the PWS ID associated with Facility or Sampling Point record does not exist in CMDP, DSE displays this validation error. This can be resolved by correcting the parent PWS ID record in S/State before running the DSE again. Once the parent PWS ID is corrected the Facility and Sampling Point child records will be migrated. 4) ERROR :09:21,089 [joblaunchertaskexecutor-1] gov.epa.apps.dse.job.s2c.config.facility.facilityitemwriter [] - Error found while invoking REST. Details of error's stack trace java.lang.illegalstateexception:

9 This validation error is generated when the DSE detects multiple validation errors in the record it is trying to migrate. The dse_app.log file should have a description of the validation error and the record that caused the error. 5) ErrorMessage=FAILED_VALIDATION_ERROR:Found field errors ERROR :15:45,128 [joblaunchertaskexecutor-1] gov.epa.apps.dse.job.s2c.config.facility.facilityitemwriter [] - Error found while invoking REST. Details of error's stack trace java.lang.illegalstateexception This validation error was generated while loading Facilities. The above screenshot shows that the error was caused because of an Invalid Water System ID. DSE displayed the error because it could not find the PWS ID associated with the Facility record. This can be resolved by correcting the parent PWS ID record in S/State before running the DSE again. Once the parent PWS ID is corrected the Facility child record will be migrated. 6) IllegalStateException: smppnttyperefid:invalid Sampling Point Type This validation error was generated while loading Sampling Points. The above screenshot shows that the error was caused because of Invalid Sampling Point Type, which can be resolved by correcting the Sampling point type for the selected record in the primacy agency s database. 7) IllegalStateException: facilityid:invalid Facility Id. 12 This validation error was generated while loading Sampling Points. The above screenshot shows that the error was caused because of Invalid Facility ID and WaterSystem ID. DSE displayed the error because it

10 could not find the PWS ID and Facility ID associated to the Sampling Point record. This can be resolved by correcting the parent PWS ID and Facility ID record in S/State before running the DSE again. Once the parent PWS ID and Facility ID records are correct the child Sampling Point record will be migrated. 8) ) ERROR :18:14,072 [joblaunchertaskexecutor-1] gov.epa.apps.dse.job.s2c.config.pws.pwsitemwriter [] - Error found while invoking REST. Details of error's stack trace java.lang.illegalstateexception: Response error::500-internal Server Error This validation error could be generated for many reasons. The most common reason is because a duplicate record was migrated. Contact CMDP Zendesk if you experience this issue. Fatal Errors: 1) IllegalStateException: Stream is already initialized. Close before re-opening. This fatal error occurs when DSE starts a new Job while still processing an existing Job (S/State to CMDP), an error which can be resolved by scheduling the jobs farther apart so that they don t overlap (e.g. from every day to every other days). Setting the incremental period will depend on the amount of data begin transferred. 2) Encountered an error executing step cmdp2statestep in job cmdp2statejob (Connection Timed Out) This fatal error occurs when the network connection drops while the job is still running. Restart the DSE job. Additionally, the submitter can view the CMDP Dashboard to see which jobs are only in submitted status and not accepted status. 3) Single row query generating multiple results This fatal error is caused when the parent SQL query is expecting a single record from the child SQL query, but the child SQL query is generating multiple records. This can be resolved by running the SQL query that generated the error and debugging to see which record in the primacy agency s database caused the issue,

11 and correcting it. Please refer to Checklist to Migrate State Data to CMDP document to identify this issue before running DSE. 4) ORA-08177: can't serialize access for this transaction ; nested exception is java.sql.sqlexception: ORA-08177: can't serialize access for this transaction If this fatal error is generated by the DSE, contact CMDP Zendesk for help.