Empfehlungen vom BigData Admin

Size: px

Start display at page:

Download "Empfehlungen vom BigData Admin"

Aldous Stephens
6 years ago
Views:

1 Empfehlungen vom BigData Admin an den Oracle DBA Florian Feicht, doag2017

provision of IT services focusing on and technologies in Switzerland, Germany, Austria and

2 Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems

3 With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants DÜSSELDORF Research and development budget: CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers GENEVA BERN LAUSANNE

with Oracle since 2005 Latest Projects Automation

4 Alexander Hofstetter Senior Consultant and Trainer, Trivadis GmbH, Munich Since 2008 Oracle DBA Working with Oracle since 2005 Latest Projects Automation High Availability Big Data Administration

5 Florian Feicht Team Leader, Senior Consultant and Trainer at Trivadis GmbH Focus: Oracle Database Administration Linux Administration Big Data Administration

6 NEXT by GEN Austausch Engagement Spaß Community

7 Agenda 1. Big Data vs. Oracle Database 2. Automation 3. Monitoring Monitoring - Performance Monitoring - Logging 4. Conclusion

8 Big Data vs. Oracle Database

9 What is Big Data? Volume Variety Velocity

10 Tasks in Big Data administration Handle huge amount of servers Handle huge amount of products Take care about High Availability Take care about security issues Work very close with the developers

12 Tasks in Big Data administration Handle huge amount of servers Usually more then 100+ systems Handle huge amount of products Usually more then 10+ tools

13 Tasks in Big Data administration Work very close with the developers A DevOps culture is a must Very few admin only tasks Bash/Java/Python Knowhow

14 Key factors for success Working in DevOps mode Automate everything Centralized monitoring Performance data collection Logfile monitoring/collection

16 Transfer the success factors Challenges are very well known and quite similar Different tools and methods are used Let s see how to transfer the stuff

17 Automation

18 Automation - Why Raising Efficiency Standardization Cost Reduction

19 Automation - Preparation Document all Tasks / Workflows Setup Environment Maintenance Support Tasks for App / Customers Which workflows can be automated?

21 Automation - How Scripting Bash Perl Python Tools

22 Automation Oracle Example hosts: test-db allow_duplicates: yes roles: - role: shutdown_db - role: yum_update_all - role: restart_server - role: relink_all - role: gi_relink - role: restart_server - role: startup_db

23 Automation Oracle Example name: get RDBMS Home shell: awk 'BEGIN{ FS=":" }!/^#.*/ && /.*:.*:.*/ { print $2 }' /etc/oratab grep -i rdbms sort -u register: oracle_home - name: relink RDBMS become: true become_user: oracle shell: export ORACLE_HOME={{ item }}; $ORACLE_HOME/bin/relink all with_items: - "{{ oracle_home.stdout_lines }}"

24 Monitoring

25 Monitoring - Challenges Monitor (almost) everything Use central repositories Keep the data as long as possiblle Find the proper tools for each purpose Visualize results

26 Monitoring - Performance

27 Performance and Capacity Management Centralized gathering of workload information Automated collection routines Central analysis repository Establishing a centralized GUI for various purpose Ability for performance and capacity forecasting (Machine Learning)

28 Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI

29 Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI

30 Collectd Open Source and free of charge Written in C Gathers metrics from different sources Stores this information or distribute it over the network Over 100 plugins Oracle database

31 Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI

32 OpenTSDB Open Source and free of charge Time series database schema Based on Hbase Time series data point: Metric name Unix timestamp Value Tags

33 Example Collectd Datensammler OpenTSDB Time Series DB Grafana GUI

34 Grafana Open Source and free of charge Visualize metrics Very easy dashboard creation Time series analytics Many plugins for different data sources

35 Demo

36 Monitoring - Logging

37 Monitoring - Logging Centralize Logging for all useful logs Combine administrator logs with developer logs Why is Logging Monitoring Important Get the big picture after Errors / Failures

38 Monitoring - Logging

39 Logstash Open Source serverside data collector Designed to send data to elastic search INPUT TRANSFORM OUTPUT

40 Elasticsearch REST based searchengine Apache Lucene is used for indexing and searching Stores search results in JSON Very fast and scalable

41 Kibana Open Source analytics and visualization platform Designed to work with Elasticsearch Dynamic Dashboards

42 Demo

43 Conclusion

44 Conclusion Challenges are similar Be open minded for new tools Dashboards are good for a quick overview Work close together with the developers

Florian Feicht Alexander Hofstetter Twitter: @FlorianFeicht florian.

45 Florian Feicht Alexander Hofstetter

Trivadis @ DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how!

46 DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit

Cloud Acceleration. Performance comparison of Cloud vendors. Tobias Deml DOAG2017

Cloud Acceleration. Performance comparison of Cloud vendors. Tobias Deml DOAG2017 Performance comparison of Cloud vendors Tobias Deml Consultant @TobiasDemlDBA DOAG2017 About Consultant, Trivadis GmbH, Munich Since more than 9 years working in Oracle environment Focus areas Cloud Computing