Extending the BOSH Backup and Restore Framework. Therese Stowell, Product Manager Chunyi Lyu, Engineer Platform Recovery Team, Pivotal

Transcription

1 Extending the BOSH Backup and Restore Framework Therese Stowell, Product Manager Chunyi Lyu, Engineer Platform Recovery Team, Pivotal

2 What is the Data in Cloud Foundry Cloud Foundry Data service Cloud Foundry Database Blobstore BOSH Database Blobstore Application data Data service BOSH

3 What can go wrong? Software failure a failed upgrade user error security issues, e.g. ransomware! Hardware failure datacenter failure Planned hardware upgrade

4 BOSH Backup and Restore Framework The contract: BBR cli orchestrates Individual component implements backup and restore scripts UAA Cloud Controller cf-networking Enabling components backup and restore scripts Backup-and-Restore-SDK-release Test frameworks Disaster-Recovery-Acceptance-Tests Pivotal-Disaster-Recovery-Acceptance-Tests

5 BOSH Backup and Restore Workflow BBR orchestrates Triggers backup and restore scripts on the jobs in the deployment Copies the artifacts to/from the jumpbox Each release implements bbr scripts Creates BBR scripts as appropriate Takes / puts the artifact from the contractual location Backups / restores any BOSH deployment or director BBR is a cli tool jumpbox 3 deployment 1 transfer artifact from deployment to jumpbox CF Operator bbr backup <deployment> bbr 2 Backup scripts: backup-lock backup backup-unlock Call backup scripts SDK Backup / Restore of: * MySQL and Postgres DBs * External blobstores

6 bbr backup cf-deployment CF Operator 1 bbr backup director BOSH B+R scripts backup 1 bbr backup cf-deployment bbr Jumpbox 2 3 call scripts across jobs in order: - pre-backup-lock - backup - post-backup-unlock transfer artifacts from deployments to jumpbox Director VM CF UAA Credhub releases* blobstore B+R scripts B+R scripts B+R scripts B+R scripts backup backup pre-backup-lock backup post-backup-unlock backup *releases: CAPI UAA Routing Credhub CF-Networking NFS Volume services Notifications

7 What does the contract give us? Contract Benefits Release authors are responsible for developing and maintaining backup Scripts are packaged in their releases, and so are always compatible with the releases Workflow is consistent across different infrastructures Applicable for all BOSH deployments Back up is triggered at a logical level rather than a hardware level Data can be flushed Locking enables consistency across data stores Backup / restore can be smart A backup script can choose not to back up all the data, alter the data in some form, add metadata, encrypt A restore script can regenerate credentials, decrypt, alter the data Data transfer decoupled from backup enables less downtime

8 Test Framework DRATS BBR *-DRATS UAA SAMPLE TESTCASE P-DRATS 1 Trigger before-backup for all testcases 1 Create user foo 2 bbr backup <deployment> 3 Trigger after-backup for all testcases 3 Delete user foo 4 bbr restore <deployment> 5 Trigger after-restore for all testcases 5 Confirm that user foo exists

9 Disaster Recovery Options Disaster Recovery Restore the PCF Data Recreate the PCF Data Preconditions IaaS prepared for PCF install IaaS prepared for PCF install Steps 1. Recreate PCF 2. Restore 3. Apply changes to make restored PCF match the other active PCF 1. Recreate PCF 2. Trigger automation to recreate orgs, spaces, etc. 3. Notify app developers to repush apps, recreate service instances and bindings RTO (Recovery Time Objective) Platform Time to recreate PCF Time to recreate PCF Apps Time to restore Time until orgs/spaces/etc have been recreated + apps have been repushed RPO (Recovery Point Objective) Platform Time of the last backup Current time Apps Time of the last backup Current time

10 DR: BBR Restore vs Automate Recreation of State Using BBR to back up CCF enables : Completeness - BBR supports backing up BOSH, including releases, Credhub, UAA, and service instances created with an on demand service broker. With PCF1.12, Ops Manager export no longer includes releases. Consistency - referential integrity between the database and the blobstore, because a lock is held while both the database and blobstore are backed up. Correctness - using the BBR flow addresses c2c and routing issues that can occur during restore Automating every state change is hard : App push, restage, scale Org, space, user Service instance Service bindings Routes Security groups Quotas Credentials It tends to be things like quotas being raised; or "bleeding edge" settings that have been enabled that can't be automated by tooling like Terraform ( yet ); changes that were made in the heat of an "incident"; that didn't make it back into the automation. - Pivotal s Head of Cloudops

11 What s New CF Operator The contract: BBR cli A Reduced downtime Deployment scripts B Cf-deployment ops files External databases + TLS External blobstores + TLS PKS API SDK C jumpbox 1 3 transfer artifact from deployment to jumpbox bbr backup <deployment> bbr 2 A Call backup scripts - External databases + TLS - External blobstores + TLS B-DRATS deployment Backup scripts: backup-lock backup backup-unlock B Backup and Restore SDK Bosh Release Backup / Restore of: * MySQL and Postgres DBs * External blobstores C

12 BBR CF Operator BBR Cli parallelization A 1 bbr backup <deployment> Reduced downtime Faster artifact transfer to / from VMs jumpbox bbr A transfer artifact from deployment to jumpbox in parallel 2 Call backup scripts in parallel deployment Backup scripts: backup-lock backup backup-unlock B Backup and Restore SDK Bosh Release Backup / Restore of: * MySQL and Postgres DBs * External blobstores C

13 External Databases SDK: MariaDB MySQL Postgres 10.1.x 5.5.x, 5.6.x, 5.7.x 9.4.x, 9.6.x Including TLS support SDK - Used by cf-deployment - Will be used by BOSH and Credhub

14 BBR Backup of Versioned S3 blobstore Versioning is turned on and buckets are replicated 1 CF S3 blobstore Backup generates a list of the current blobs B+R scripts 2 backup Backup artifact (list of the current blobs) Transferred Enabled by use of the Backup Restore SDK BOSH Release Minimizes API downtime and network traffic Jumpbox can stay small Bucket replication Bucket retention S3 blobstore S3 blobstore replicant

15 BBR Backup of unversioned S3 blobstore With bucket being copied within the infrastructure 1 CF S3 blobstore Backup triggers copy of blobs from one blobstore to another 2 B+R scripts backup Backup artifact (metadata of backup bucket) Transferred Enabled by use of the Backup Restore SDK BOSH Release Copy done in parallel to minimize API downtime Minimizes network traffic Jumpbox can stay small Ceph blobstore Ceph backup

16 In Progress - Adding support for versioned GCS and Azure blobstore

17 Upcoming Backup artifact validation Backup artifact encryption BBR support - deployment scripts PKS on demand clusters PCF Redis

18 Backup Artifact Validation Coming soon!

19 Partners - Redislabs - BOSH deployed clustered Redis - Stark & Wayne - Shield + BBR - Dell/EMC - DataDomain integration - CrunchyData - Postgres

20 Summary Done - External blobstores and databases, with TLS - Testing framework for BOSH - parallelization In progress - Azure and GCP blobstore Upcoming - Artifact validation - Artifact encryption

21 Questions? Github: Slack: cloudfoundry.slack.com #bbr Office Hours: 3:55pm, Thursday 19 April, Lounge 1 Chunyi: clyu@pivotal.io

22