Empfehlungen vom BigData Admin an den Oracle DBA Florian Feicht, Alexander Hofstetter @FlorianFeicht @lxdba doag2017
Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems. 2 23.11.2017
With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants DÜSSELDORF Research and development budget: CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers GENEVA BERN LAUSANNE 3 23.11.2017
Alexander Hofstetter Senior Consultant and Trainer, Trivadis GmbH, Munich Since 2008 Oracle DBA Working with Oracle since 2005 Latest Projects Automation High Availability Big Data Administration 4 23.11.2017
Florian Feicht Team Leader, Senior Consultant and Trainer at Trivadis GmbH Focus: Oracle Database Administration Linux Administration Big Data Administration 5 23.11.2017
NEXT by GEN Austausch Engagement Spaß Community
Agenda 1. Big Data vs. Oracle Database 2. Automation 3. Monitoring Monitoring - Performance Monitoring - Logging 4. Conclusion 7 23.11.2017
Big Data vs. Oracle Database 8 23.11.2017
What is Big Data? Volume Variety Velocity 9 23.11.2017
Tasks in Big Data administration Handle huge amount of servers Handle huge amount of products Take care about High Availability Take care about security issues Work very close with the developers 10 23.11.2017
11 23.11.2017
Tasks in Big Data administration Handle huge amount of servers Usually more then 100+ systems Handle huge amount of products Usually more then 10+ tools 12 23.11.2017
Tasks in Big Data administration Work very close with the developers A DevOps culture is a must Very few admin only tasks Bash/Java/Python Knowhow 13 23.11.2017
Key factors for success Working in DevOps mode Automate everything Centralized monitoring Performance data collection Logfile monitoring/collection 14 23.11.2017
15 23.11.2017
Transfer the success factors Challenges are very well known and quite similar Different tools and methods are used Let s see how to transfer the stuff 16 23.11.2017
Automation 17 23.11.2017
Automation - Why Raising Efficiency Standardization Cost Reduction 18 23.11.2017
Automation - Preparation Document all Tasks / Workflows Setup Environment Maintenance Support Tasks for App / Customers Which workflows can be automated? 19 23.11.2017
20 23.11.2017
Automation - How Scripting Bash Perl Python Tools 21 23.11.2017
Automation Oracle Example --- - hosts: test-db allow_duplicates: yes roles: - role: shutdown_db - role: yum_update_all - role: restart_server - role: relink_all - role: gi_relink - role: restart_server - role: startup_db 22 23.11.2017
Automation Oracle Example --- - name: get RDBMS Home shell: awk 'BEGIN{ FS=":" }!/^#.*/ && /.*:.*:.*/ { print $2 }' /etc/oratab grep -i rdbms sort -u register: oracle_home - name: relink RDBMS become: true become_user: oracle shell: export ORACLE_HOME={{ item }}; $ORACLE_HOME/bin/relink all with_items: - "{{ oracle_home.stdout_lines }}" 23 23.11.2017
Monitoring 24 23.11.2017
Monitoring - Challenges Monitor (almost) everything Use central repositories Keep the data as long as possiblle Find the proper tools for each purpose Visualize results 25 23.11.2017
Monitoring - Performance 26 23.11.2017
Performance and Capacity Management Centralized gathering of workload information Automated collection routines Central analysis repository Establishing a centralized GUI for various purpose Ability for performance and capacity forecasting (Machine Learning) 27 23.11.2017
Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI 28 23.11.2017
Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI 29 23.11.2017
Collectd Open Source and free of charge Written in C Gathers metrics from different sources Stores this information or distribute it over the network Over 100 plugins Oracle database 30 23.11.2017
Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI 31 23.11.2017
OpenTSDB Open Source and free of charge Time series database schema Based on Hbase Time series data point: Metric name Unix timestamp Value Tags 32 23.11.2017
Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI 33 23.11.2017
Grafana Open Source and free of charge Visualize metrics Very easy dashboard creation Time series analytics Many plugins for different data sources 34 23.11.2017
Demo 35 23.11.2017
Monitoring - Logging 36 23.11.2017
Monitoring - Logging Centralize Logging for all useful logs Combine administrator logs with developer logs Why is Logging Monitoring Important Get the big picture after Errors / Failures 37 23.11.2017
Monitoring - Logging 38 23.11.2017
Logstash Open Source serverside data collector Designed to send data to elastic search INPUT TRANSFORM OUTPUT 39 23.11.2017
Elasticsearch REST based searchengine Apache Lucene is used for indexing and searching Stores search results in JSON Very fast and scalable 40 23.11.2017
Kibana Open Source analytics and visualization platform Designed to work with Elasticsearch Dynamic Dashboards 41 23.11.2017
Demo 42 23.11.2017
Conclusion 43 23.11.2017
Conclusion Challenges are similar Be open minded for new tools Dashboards are good for a quick overview Work close together with the developers 44 23.11.2017
Florian Feicht Alexander Hofstetter Twitter: @FlorianFeicht florian.feicht@trivadis.com Twitter: @lxdba alexander.hofstetter@trivadis.com 45 23.11.2017
Trivadis @ DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit 46 23.11.2017