A Few Words about the Speaker OpenEdge MWUG Spring 2016 Nectar Daloglou, White Star Software nd@wss.com Nectar Daloglou; Progress & QAD since 2000. Performed specialized services at more than 50 QAD customer sites: Progress Administration Install/Upgrades/Migrations of Progress and QAD Applications Technical Audits / Performance Tuning Business Continuity Strategies Now part of White Star Software Key Member in large Progress and QAD implementation 2 Agenda Introduction Business Continuity What is OpenEdge? Why use OpenEdge? Requirements Setting up OpenEdge Using OpenEdge Alternatives Questions Before We Start Your Mileage May Vary Information in this presentation may not necessarily apply to your environment An overview of OpenEdge, not a complete how-to guide Meant to provide some useful information on OpenEdge to help incorporate in your Business Continuity Plan Business Continuity High Availability -More Localized Failure Business Continuity Disaster Recovery -Natural or human induced disaster -More serious Business Continuity Myths Disasters are rare occurrences A business continuity plan is only made to prepare for a natural catastrophe Disaster recovery sites are rare The disaster recovery site must be situated thousands of miles away is limited 1
Question How many times in the last five years have you had to invoke a Business Continuity Plan? What were the causes of the invocations? : Forrester/Disaster Recovery Journal Business Continuity Preparedness Survey, Q4 2014 : Forrester/Disaster Recovery Journal Business Continuity Preparedness Survey, Q4 2014 Downtown Calgary Flood Power Outage : http://www.flickr.com/photos/ryan_quan/9147845946/ OpenEdge OpenEdge helps in recovering from a disaster by replicating the Progress databases to another server or site. What is OpenEdge? Product from Progress Software that provides (near) real-time synchronization between two or more databases. server on the source machine The replication server reads after image notes upon request of the agent agent(s) on target machine(s) The agent(s) apply after image notes to the target database Limits/Features: One source (single direction replication) Up to 2 target databases can be used for reporting with additional licensing Picture : Gravic.com 2
OpenEdge Architecture Why do OpenEdge? Easier than a job search Near real-time replication of data Only supported method to do near real-time replication Agent Requirements machine(s) with same: Operating system version Patches Same Kernel parameters Two Reasons Nothing unexpected when you fail over Easier to maintain if both machines are identical Disk Requirements Two aspects of disk capacity Storage capacity How much stuff you can store Throughput capacity How much stuff you need to quickly access Both machines need the same storage capacity (DB, application, temp files, ) Disk throughput can affect performance in the event of a failover but you can choose to run only core services in failover mode CPU & Memory Best case is if and machines are exactly the same If this is not financially possible then Less cores (faster if possible) on replication machine More cores for replication with reporting Most cores in Production Do you need to support ALL users or only core services You can have less memory if you are going to reduce usage Estimating network bandwidth AID x AIRM = AIDR AIDR x AIRO = AIDRO AIDRO X (PRD) = AIDROP AID: The total size of all the after-image blocks. AIRM: The value 1.5; the multiplier for additional afterimage notes due to OpenEdge. AIDR: The number of after-image blocks with OpenEdge. AIRO:The value 1.1; the multiplier for OpenEdge overhead. AIDROP: The number of after-image blocks with OpenEdge and overhead. (PRD): The period of time : http://documentation.progress.com/ 18 3
Sample bandwidth estimate Suppose we have five databases that fill up 360 MB of AI files during a peak hour Additional AI notes for = 1.5 * AI Size 1.5 * 360 MB = 540 MB Overhead = 1.1 * AI Notes 1.1 * 540 MB = 594 MB Bandwidth per second: 594 MB/60/60=0.165 MB/second In kb/s: 0.165 MB/s * 1024 = 168.96 KB/s required for data transfer Or 1,351.68 kb/s (1 kilobyte = 8 kilobits) Non-DB What really makes up you application? QAD Application files User files Other applications (Apache, Tomcat, 3 rd party applications, ) Options: Remote mirrors (hardware) Operating system commands (Example: rsync) Clustering Any others? : http://documentation.progress.com/ Initial Setup: OpenEdge (Offline) After imaging must be enabled for the source database With the database down you can enable site replication for the source database proutil <dbname> C enablesitereplication source Move the properties files into place; Currently the same directory as the DB Start the source database with -DBService replserv and - pica parameters Do an online backup of the source Restore the backup to the target Initial Setup: OpenEdge (Online) Enable After Imaging (If necessary) Put the replication properties files in place You need to use deferred agent startup [server] defer-agent-startup=<number-of-minutes> Note: number of minutes must be between 0 and 10080 (1 week) Enable replication proutil <dbname> -C enablesitereplication source Do an online backup to build target database probkup online <source-db> <bkup_file> -REPLCreation Initial Setup: OpenEdge (continued) After the restore to the target is complete Enable site replication on the target proutil <dbname> C enablesitereplication target Or prorest <dbname> <backupname> -REPLTransition Start the target with -DBService replagent and -S <service or port number> parameters Use the dsrutil utility to monitor replication dsrutil <dbname> C monitor Or look for the following message in the target database.lg file The and databases are synchronized. (10668). -pica Determines the size of the interprocess communication queue between the RDBMS and the replication server Default value 64k in 10.1C+ and 6k before Default value is generally too low Always set it to it s maximum value In 10.1A+ set it to 8192 Prior to 10.2B08+ set it up to 1000000 Pica Boo! 4
Calculating -pica Property Files -pica = (Total AI Writes/Block count) * 1.25 Total AI writes can be determined with promon promon R&D Option 2 (Activity) Option 6 (AI Log) *Note: Gather Total AI writes at the busiest update portion of the day Block count in version 10+ = 9.16 When in doubt set it larger (memory is cheap) DB <DB>.repl.properties DB <DB>.repl.properties 26 Parameter Examples Sample Property File Manual or automatic transition Transition to source or to normal DB TCP/IP Port Timeouts Secondary site definition 27 [server] control-agents=agent1 database=source transition=manual transition-timeout=600 defer-agent-startup=10080 agent-shutdown-action=recovery [control-agent.agent1] name=agent1 database=qadprd01 host=lov9100 port=7990 connect-timeout=120 replication-method=async critical=0 [transition] database-role=normal [agent] name=agent1 database=target listener-minport=4000 listener-maxport=4500 [transition] database-role=normal Hint: The source and target repl.properties file can be merged for ease of use in the case a transition 28 Troubleshooting: Before Failure Troubleshooting: After Failure DB DOWN AI LOCKED Agent Connection Failure Agent 29 30 5
Common Failures All AI files are in a Locked state #1 Cause: The is unable to reach the Parameter in OE Property File: connect-timeout=86400 : Specifies for how long, in seconds, the server will attempt to connect to its configured agents. (86400 seconds = 24 hours) Monitoring Use the _Repl VSTs Use the dsrutil utility with the monitor option dsrutil <dbname> -monitor This can be done on the source or target side ProTop Status option displays status code dsrutil <dbname> -status 31 32 Monitoring with VSTs Monitoring with dsrutil: _Repl- This provides information about the replication server _Repl-AgentControl This provides information about the replication agents that the server is controlling _Repl-Agent This provides information about the replication agent 33 Normal Processing Sample: OpenEdge Monitor Page 1 : /db/prod/prod1 is enabled as OpenEdge : is: In Normal Processing Number of configured agents: 1 Delay Interval (current / min / max): 345 / 5 / 500 Recovery information: State: No recovery being performed Agents needing recovery: 0 Agents connected: 0 Agents in synchronization: 0 Transition information: Type: Manual 34 Monitoring with dsrutil: Agent OpenEdge Monitor Page 1 : /db/prod/prod1 is enabled as OpenEdge : Agent: Name: agent1 ID: 1 Host name: 204.19.29.11 State: Normal Processing Ready: Yes Critical: No Method: Asynchronous Agent is waiting for: Nothing Maximum bytes in TCP/IP message: 16696 /Agent connection time: Wed May 6 03:01:15 2015 Delay Interval (current / min / max): 40 / 5 / 500 35 Monitoring with dsrutil: Agent con t AI Block Information: RDBMS Block (Seq / Block): 7 / 7801 Last Processed Block (Seq / Block): 7 / 7800 Latency Information: Repl behind DB by: 14 second(s) Current Transaction: 309904965 Last Transaction Applied to : 309904960 Current as of (, ): Fri May 8 22:07:44 2015 36 6
Monitoring with ProTop Failover Primary database fails If primary machine s AI files are available apply last transactions dsrutil <dbname> -C applyextent <extentname> Then transition the database dsrutil <dbname> -C transition [failover] [failover] will transition the target via the replication agent then transition the source. If replication is not running you cannot use the failover option 37 38 Failure Scenario in Pre- Transition State DB DOWN AI LOCKED AI Extents (if Available) AI Roll Forward Connection Failure Agent Agent 39 40 Becomes (or Normal ) Failback Configure original Primary machine to be the target No activity on the databases No users on the database Run dsrutil <dbname> -C transition failover This will transition the primary database (now target) to transition to a source and then transition the secondary database (now a source) to a target. If you have restart-after-transition set in the properties files then the databases will be restarted in their proper roles 41 42 7
Failback: Role Reversal Failback: Role Reversal Agent Agent 43 44 Alternatives to OpenEdge Hardware base replication Log-based replication (After image replication) Trigger-based replication Hardware replication Also known as remote mirroring Easy to implement Changes the ownership of support from Progress to the hardware vendor It is expensive Does not support point in time restore Log-based Also know as asynchronous (or poor man s) replication Not true replication but a point in time synchronization between two machines After image files are taken from the source machine and applied throughout the day Pros: Cheap Cons: Not synchronous Must support and maintain scripting yourself unusable while being replicated Trigger-based replication Adding schema triggers to implement the replication of data You can replicate exactly what you want Very difficult to support and maintain 8
Things to remember OpenEdge is the only way to keep a second copy of your production data in sync at all times OpenEdge requires advanced thought and planning Existing recovery plan including after image Hardware and infrastructure planning OpenEdge requires monitoring and maintenance in an ongoing basis There are costs involved with OpenEdge Hardware Software Implementation and training Tips Start with a good backup plan You must have multi-volume AI Use the AI management tool to archive full AI extents Remember to remove old archives Choose a replication method that makes sense for you If you choose OpenEdge use asynchronous replication for performance reasons Remember to set -pica Test extensively before doing replication in your production environment Questions Questions? Questions or comments? Feel free to e- mail me: Nectar Daloglou: nd@wss.com Thank You! Try ProTop Free The #1 FREE OpenEdge Monitoring Tool http://dbappraise.com/protop.html 53 9
White Star Software The oldest and most respected independent DBA consulting firm in the world Four of the world s top OpenEdge DBAs Author of ProTop, the #1 FREE OpenEdge Monitoring Tool http://dbappraise.com/protop.html 10