13.11.2018 Trove Onboarding Session Introductory course for contributors and reviewers Bartosz Żurkowski Cloud and Big Data Engineer, Samsung R&D Institute Poland IRC: bzurkowski EMAIL: b.zurkowski@samsung.com Przemysław Godek Senior Cloud and Big Data Engineer, Samsung R&D Institute Poland IRC: pgodek EMAIL: p.godek@partner.samsung.com
Outline Introduction Trove mission Features overview Datastore support Deep dive How do I get started? Development environment setup Building guest images Guest customization via cloudinit Where contribution is needed? Overall architecture Handling datastore diversity using strategy pattern
What is Trove? Database as a Service for OpenStack Provides full database lifecycle management Provisioning, configuration, backups, scaling Multi-datastore support 11 database engines Relational, non-relational Single-instance and clustered deployments Unified management interface Built entirely on OpenStack Synergy of Nova, Cinder, Swift, Glance and Neutron Application optimization DB performance tuning Replication and clustering Scaling Periodic backups DB software upgrades Hardening DB software setup and config Virtual resource provisioning DBA responsibilities DBaaS (Trove) responsibilities
Features overview Instance provisioning Instance resizing (volume, flavor) Database and user management Configuration groups Backups (full, incremental, scheduled) Cluster provisioning Cluster sizing (grow, shrink) Replication setup Replication failover (promote read replica, eject source) Datastore upgrades Logs (guest, database) Security groups management Flavors management
Datastore capability matrix Did you find your favourite database on the slide? Datastore Type Provisioning Schema management Backup and restore Resizing Replication Clustering MySQL MariaDB PostgreSQL Relational Percona DB2 Redis Key-value - Cassandra - Column Vertica - MongoDB - Document CouchDB - Couchbase Multi-model -
Let s dive into the internals
Deliverables Trove openstack/trove Python Trove Client openstack/python-troveclient Trove Dashboard openstack/trove-dashboard Trove Specs openstack/trove-specs Trove Tempest Plugin openstack/trove-tempest-plugin
Overall architecture DynamoDB RedShift Trove RDBMS NoSQL RDS Elasticache DW EC2 S3 Nova Swift EBS VPC Cinder Neutron
Overall architecture Trove API Exposes REST API for end user Handles simple tasks Trove Task Manager Handles heavy database operations enqueued by API (provisioning, backups, scaling) Trove Guest Agent Exposes abstract interface for interacting with the database service Trove Conductor Serves as a proxy for Guest Agent to communicate back with the control plane
Overall architecture: Performing database backup 1 4 2 3 5 9 8 6 7 1. User hits Trove API via CLI or Horizon 2. API creates initial backup record in the database 3.... queues backup job onto the message bus 4. and responds to the user 5. Taskmanager grabs job from message bus 6. and asks guestagent to perform backup 7. Guestagent performs backup and streams output to Swift 8. then reports backup completion back to Conductor 9. Conductor updates backup status in the database
How to handle datastore diversity? Configuration Procedures Formats (INI, YAML, JSON) Directories Sets of parameters Backup and restore Procedures Backuping tools Clustering Procedures Primitives (Cassandra tokens, Redis slots) Management tools (nodetool, redis-cli) Replication Storage...
with Strategy pattern Behavioral software design pattern Enables selecting an algorithm at runtime Code receives run-time instructions of which algorithms to use Depending on datastore type, strategy implements a specific backup, replication, storage or clustering routine Locations in code: guestagent/strategies/* backup, restore, replication guestagent/datastore/experimental/* database service management common/strategies/cluster/experimental/* - cluster management
Strategy pattern examples: Database service DatabaseService prepare() list_databases() create_database() list_users() create_user() get_status() fetch_logs() MariaDBService prepare() list_databases() create_database()... RedisService prepare() list_databases() create_database()... CassandraService prepare() list_databases() create_database()...
Strategy pattern examples: Clustering ClusteringStrategy create_cluster() grow_cluster() shrink_cluster() upgrade_cluster() GaleraClusteringStrategy create_cluster() grow_cluster() shrink_cluster() upgrade_cluster() RedisClusteringStrategy create_cluster() grow_cluster() shrink_cluster() upgrade_cluster() CassandraClusteringStrategy create_cluster() grow_cluster() shrink_cluster() upgrade_cluster()
Determining strategy for datastore Loading strategy Strategy configuration
How do I get started?
Development environment setup: Devstack Download Devstack git clone git://git.openstack.org/openstack-dev/devstack.git && cd devstack Edit local.conf [[local localrc]]... enable_plugin trove git://git.openstack.org/openstack/trove LIBS_FROM_GIT=python-troveclient Run Devstack utility./stack.sh Trove sources reside in /opt/stack/trove Useful documentation: https://github.com/openstack/trove/tree/master/devstack https://docs.openstack.org/devstack/latest/configuration.html
Development environment setup: Kolla Ansible Install Kolla Ansible: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html Edit /etc/kolla/globals.yaml enable_trove: "yes" enable_cinder: "yes" enable_cinder_backend_lvm: "yes" # required setting LVM volumes group trove_dev_mode: "yes" Run deployment kolla-ansible -i all-in-one prechecks kolla-ansible -i all-in-one deploy Trove sources reside in /opt/stack/trove
Building guest images Install Disk Image Builder: https://docs.openstack.org/diskimage-builder/latest/user_guide/installation.html Clone Trove repository git clone https://github.com/openstack/trove.git Run build-image command./trove/integration/scripts/trovestack build-image mysql Guest image will be built into $HOME/images directory Useful documentation: https://docs.openstack.org/trove/latest/admin/building_guest_images.html
Load images into Trove Upload image to Glance openstack image create --file $HOME/images/<image-name>.qcow2 \ --disk-format qcow2 --public ubuntu_mysql Setup datastore in Trove trove-manage datastore_update mysql "" trove-manage datastore_version_update mysql 5.7 mysql <glance-image-id> "" 1 trove-manage datastore_update mysql 5.7 trove-manage db_load_datastore_config_parameters \ mysql 5.7 \./trove/trove/templates/$datastore_type/validation-rules.json
Cloudinit scripts Allow to customize VM on the first boot Each datastore defines its own <datastore-name>.cloudinit script in /etc/trove/cloudinit directory Task Manager collects cloudinit content and injects it into VM via user data Example use cases Setting authorized keys for connection from the host Synchronizing guest code with host or external repository Configuring monitoring agents Running Ansible playbooks
Where contribution is needed?
Appealing work awaiting Python 3 support Aligning OpenStack CLI with Trove CLI Cluster management (backups, rolling upgrades, flavor resizing) Datastore support enhancement (Oracle, SQL Server, HBase, Druid, etc.) Cluster status monitoring Backup encryption Check out full list of our ideas https://blueprints.launchpad.net/trove
Meet core contributors & community members With whom could I chat about Trove at the conference? Bartosz Żurkowski, b.zurkowski@samsung.com Przemysław Godek, p.godek@partner.samsung.com Check out the ongoing work: https://etherpad.openstack.org/p/trove-general You are invited to test Trove, subsequently share your bug fixes, and enhance the documentation Weekly meetings: Wed 14:00 UTC on IRC #openstack-meeting-alt channel, up-to-date schedule: https://wiki.openstack.org/wiki/meetings/trovemeeting Obtain help at Trove IRC channel: #openstack-trove: http://webchat.freenode.net/?channels=openstack-trove
Questions & Answers openstack @OpenStack openstack OpenStackFoundation
Thank You! openstack @OpenStack openstack OpenStackFoundation