Azure Cosmos DB NoSQL Migration

Size: px

Start display at page:

Download "Azure Cosmos DB NoSQL Migration"

Dorthy Mae Barber
5 years ago
Views:

1 Azure Cosmos DB NoSQL Migration

2 A P P D E V E L O P E R S FA C E N E W C H A L L E N G E S Todays applications place new demands and data requirements on app developers. NoSQL developers increasingly look to leverage cloud capabilities and fully-managed services, so they can focus on application innovation. Respond to rapid, variable scale and demand at global scale Deliver customer analytics and business insights Power real-time experiences Meet customers high performance expectations

3 NOSQL DEVELOPERS WANT ENTERPRISE -GRADE MANAGED SERVICES FOR NOSQL DATABASES Tedious set-up and maintenance requirements Expensive and complex HA solutions Challenges with NoSQL IaaS/onpremises offerings: Achieving end-to-end security Over-provisioning of resources to achieve scale Unpredictable costs

4 A Z U R E C O S M O S D B A globally distributed, massively scalable, multi-model database service SQL MongoDB Table API Key-value Column-family Document Graph Elastic scale out of storage & throughput Guaranteed low latency at the 99 th percentile Five well-defined consistency models Turnkey global distribution Comprehensive SLAs

5 M O D E R N I Z E A N Y N O S Q L A P P W I T H A Z U R E C O S M O S D B AZURE COSMOS DB OFFERS NOSQL APP DEVELOPERS: Fully-managed database service, born in the cloud Turnkey global distribution Low latency at any scale anywhere in the world Azure Cosmos DB Easy lift-and-shift migration from MongoDB and Cassandra Use existing tools, code, and drivers Automatic Indexing and partitioning Azure Cosmos DB: API for MongoDB API for Cassandra Enterprise-grade SLAs on high availability and latency Native integrations for serverless architecture, advanced analytics, and event processing Optimized for real-time scenarios NoSQL workloads

6 Azure Cosmos DB VA L U E P R O P O S I T I O N Azure Cosmos DB: API for MongoDB API for Cassandra NoSQL workloads

7 T U R N K E Y G L O B A L D I S T R I B U T I O N PUT YOUR DATA WHERE YOUR USERS ARE Automatically replicate all your data around the world, and across more regions than Amazon and Google combined. Available in all Azure regions Manual and automatic failover Automatic & synchronous multi-region replication

8 E L A S T I C S C A L E O U T O F S T O R A G E A N D T H R O U G H P U T SCALES AS YOUR APPS NEEDS CHANGE Independently and elastically scale storage and throughput across regions even during unpredictable traffic bursts with a database that adapts to your app s needs. Elastically scale throughput from 10 to 100s of millions of requests/sec across multiple regions Support for requests/sec for different workloads Pay only for the throughput and storage you need

9 G U A R A N T E E D L O W L AT E N C Y PROVIDE USERS AROUND THE WORLD WITH FAST ACCESS TO DATA Serve <10 ms read and <15 ms write requests at the 99th percentile from the region nearest to users, while delivering data globally.

10 F I V E W E L L - D E F I N E D C O N S I S T E N C Y M O D E L S CHOOSE THE BEST CONSISTENCY MODEL FOR YOUR APP Offers five consistency models Provides control over performance-consistency tradeoffs, backed by comprehensive SLAs. An intuitive programming model offering low latency and high availability for your planet-scale app. Strong Bounded-stateless Session Consistent prefix Eventual

11 M U LT I P L E D ATA M O D E L S A N D A P I s USE THE MODEL THAT FITS YOUR REQUIREMENTS, AND THE APIS, TOOLS, AND FRAMEWORKS YOU PREFER Cosmos DB offers a multitude of APIs to access and query data including, SQL, various popular OSS APIs, and native support for NoSQL workloads. Use key-value, tabular, graph, and document data Data is automatically indexed, with no schema or secondary indexes required Blazing fast queries with no lag MongoDB Table API SQL Key-value Column-family Document Graph

12 C O M P R E H E N S I V E S L A s RUN YOUR APP ON WORLD-CLASS INFRASTRUCTURE Azure Cosmos DB is the only service with financially-backed SLAs for millisecond latency at the 99th percentile, % HA and guaranteed throughput and consistency Latency HA Throughput Consistency <10 ms 99 th percentile % Guaranteed Guaranteed

13 H A N D L E A N Y D ATA W I T H N O S C H E M A O R I N D E X I N G R E Q U I R E D Azure Cosmos DB s schema-less service automatically indexes all your data, regardless of the data model, to delivery blazing fast queries. Automatic index management Synchronous auto-indexing No schemas or secondary indices needed Works across every data model GEEK Item Color Microwave safe Liquid capacity CPU Memory Storage Geek mug Coffee Bean mug Graphite Yes 16ox????????? Tan No 12oz????????? Surface book Gray?????? 3.4 GHz Intel Skylake Core i7-6600u 16GB 1 TB SSD

14 T R U S T Y O U R D ATA T O I N D U S T R Y - L E A D I N G S E C U R I T Y & C O M P L I A N C E Azure is the world s most trusted cloud, with more certifications than any other cloud provider. Enterprise grade security Encryption at Rest Encryption is enabled automatically by default Comprehensive Azure compliance certification

15 N O S E R V E R M A N A G E M E N T Azure Cosmos DB s service is regularly and automatically updated to support new client database versions and wire protocol formats. No need to manage or patch database servers Data does not need to be migrated to support a new protocol version MongoDB

16 Account R E S O U R C E S & C O N S U M P T I O N Container Item

17 R E S O U R C E H I E R A R C H Y CONTAINERS Logical resources surfaced to APIs as tables, collections (for documents) or graphs, which are made up of one or more physical partitions or servers. RESOURCE PARTITIONS Tables Collections Graphs Consistent, highly available, and resource-governed coordination primitives Consist of replica sets, with each replica hosting an instance of the database engine Containers Owns a set of keys (partition and row) Can uniquely belong to a tenant by assigning one partition key or one container per tenant Resource Partitions

18 C R E AT I N G A C C O U N T Account Container Item

19 C O N TA I N E R R E P R E S E N TAT I O N S Account Container = Collection Graph Table Column Family Item

20 R E Q U E S T U N I T S Request Units (RUs) is a rate-based currency Abstracts physical resources for performing requests Key to multi-tenancy, SLAs, and COGS efficiency Foreground and background activities % Memory % CPU % IOPS

21 Incoming Requests R E Q U E S T U N I T S Normalized across various access methods 1 read of 1 KB document from a single partition Each request consumes fixed RUs Applies to reads, writes, queries, and stored procedure execution Provisioned in terms of RU/sec Rate limiting based on amount of throughput provisioned Can be increased or decreased instantaneously Metered Hourly Background processes like TTL expiration, index transformations scheduled when quiescent Max RU/sec Min RU/sec Rate limit No rate limiting Replica Quiescent

22 R E Q U E S T U N I T P R I C I N G E X A M P L E Storage Cost Avg Record Size (KB) 1 Number of Records 100,000 Total Storage (GB) 100 Monthly Cost per GB $0.25 Expected Monthly Cost for Storage $25.00 Throughput Cost Operation Type Number of Requests per Second Avg RU's per Request RU's Needed Create Read Total RU/sec 900 Monthly Cost per 100 RU/sec $6.00 Expected Monthly Cost for Throughput $54.00 Total Monthly Cost [Total Monthly Cost] = [Monthly Cost for Storage] + [Monthly Cost for Throughput] = $25 + $54 = $79 per month

R E Q U E S T U N I T C A L C U L AT O R CALCULATOR The web-based calculator estimates request unit and data storage requirements for typical CRUD operations: Create Read Update

23 R E Q U E S T U N I T C A L C U L AT O R CALCULATOR The web-based calculator estimates request unit and data storage requirements for typical CRUD operations: Create Read Update Delete SAMPLE DOCUMENT The sample document is required and used to improve the accuracy of the calculator You can include more than one sample document to compare consumption requirements

24 E S T I M AT I N G M O N G O D B A P I R E Q U E S T U N I T S MEASUING MONGO THROUGHPUT The Azure Portal s metrics contain Number of requests and Request Charge charts that give an estimation of how many request units each operation is consuming in isolation or in relation to other operations. MONGO SHELL The MongoDB API supports a custom command named getlastrequeststatistics to retrive the request charge for an operation. This command can be executed using the Mongo Shell or a client SDK.

25 R E Q U E S T U N I T S P E R C O N TA I N E R FIXED-SIZE CONTAINERS Fixed-size containers have a maximum storage limit of 10 GB and a maximum throughput limit of 10,000 RU/s. 1,000,000 This type of container requires a minimum of 400 RU/s making it a good candidate for smaller development or proof-of-concept projects. UNLIMITED CONTAINERS 10,000 Unlimited containers can scale up to 1,000,000 RU/s in throughput. This type of container also requires a partition key to split data across multiple partitions. 1,000 Fixed-Size Container Unlimited Container 400 RU/s

26 C O M PAT I B I L I T Y MongoDB

27 C A S S A N D R A A P I CASSANDRA Azure Cosmos DB databases can be used as the backing database for an application written for use with Apache Cassandra using Apache licensed drivers. CQL Azure Cosmos DB has wire protocol level compatability with existing SDKs and tools that are compliant with CQLv4.

28 M O N G O A P I MONGO Azure Cosmos DB databases can be used as the data store for apps written for MongoDB. Using existing drivers, your application written for MongoDB can use Azure Cosmos DB databases instead of MongoDB databases. MongoDB In many cases, you can switch from using MongoDB to Azure Cosmos DB by simply changing a connection string. MONGODB WIRE PROTOCOL You can use existing MongoDB application code and tooling as Azure Cosmos DB implements the MongoDB 3.4 (version 5) wire protocol and supports the MongoDB aggregation pipeline.

29 M O N G O A P I A G G R E G AT I O N P I P E L I N E AGGREGATION The pipeline combines multiple aggregation stages together enabling more sophisticated queries over data sets. EXAMPLE input $match $group db.flights.aggregate([ { $match: { origin: "SEA" } }, { $group: { _id: "$destination", total: { $sum: 1 } } }, { $sort: { total: -1 } } ]); $sort output

30 A C C O U N T U R I A N D C R E D E N T I A L S Account ********.azure.com IGeAvVUp Container Item

31 A C C O U N T K E Y S MASTER KEY Master keys provide access to the all the administrative resources for the database account. Master keys: Provide access to accounts, databases, users, and permissions. Cannot be used to provide granular access to collections and documents. Are created during the creation of an account. Account ********.azure.com GTWQyPYt tzbfsokc READ-WRITE/READ-ONLY KEYS In addition to the two master keys for the Cosmos DB account, there are two read-only keys. These read-only keys only allow read operations on the account. Readonly keys do not provide access to read permissions resources. DtVqMSXu MDhIjLNb

32 A C C O U N T K E Y S - R E G E N E R AT I O N CYCLING KEYS Each account consists of two Master keys: a primary key and secondary key. Account ********.azure.com The purpose of dual keys is so that you can regenerate, or roll keys, providing continuous access to your account and data. GTWQyPYt tzbfsokc The regeneration of your keys can be performed manually using the Azure Portal or automatically using the REST API, client SDKs, Azure CLI or Azure PowerShell.

33 D ATA B A S E - L E V E L S E C U R I T Y Account Container Users Item Permissions

34 U S E R S A N D P E R M I S S I O N S USERS Cosmos DB users are associated with a Cosmos DB database. Each database can contain zero or more Cosmos DB users. Functionally, Users serve as a container for permissions. Users Permissions PERMISSIONS A Cosmos DB permission resource is associated with a Cosmos DB user. Each user may contain zero or more Cosmos DB permissions. A permission resource provides access to a security token that the user needs when trying to access a specific application resource.

35 R E S O U R C E T O K E N S RESOURCE TOKEN & PERMISSIONS Resource tokens capture the relationship between the user of a database and the permission the user has for a specific Cosmos DB resource, such as a collection or a document Azure Cosmos DB resource tokens provide a safe mechanism for allowing clients to read, write, and delete specific resources in an Azure Cosmos DB account according to the granted permissions. 1. Client device/application authenticates with the middle-tier service 2. The middle-tier service will request a resource token using a specific database-level permission 3. The database service generates a temporary-use resource token 4. The middle-tier service will return the resource token to the client 5. The client will access the resource[s] directly from Azure Cosmos DB using the resource token

36 MIGRATION I G R I O N MongoDB

37 PA R T I T I O N I N G DESIGN FOR QUOTAS Some existing applications may not have been designed with storage and throughput quotas in mind: MongoDB allows documents up to 16MB in size Azure Cosmos DB has a soft-limit of 2MB per item E MongoDB technically doesn t have a limit to the number of documents in a single, non-partitioned collection. Fixed-size containers have a storage limit of 10GB Unlimited containers do not have a storage limit but they require a partition key A B C D PARTITIONING To scale effectively with Azure Cosmos DB, you need to pick a good partition key when you create your unlimited container

38 PA R T I T I O N K E Y S Best Practices: Design Goals for Choosing a Good Partition Key Distribute the overall request + storage volume Avoid hot partition keys Partition Key is scope for [efficient] queries and transactions Queries can be intelligently routed via partition key Omitting partition key on query requires fanout Steps for Success Ballpark scale needs (size/throughput) Understand the workload Steps for Success # of reads/sec vs writes per sec Use 80/20 rule to help optimize bulk of workload For reads understand top queries (look for common filters) For writes understand transactional needs and ratio of inserts vs updates General Tips Don t be afraid of having too many partition keys Partitions keys are logical More partition keys more scalability

39 P R E - M I G R AT I O N SSL Azure Cosmos DB has strict security requirements and standards. Be sure to enable SSL when you interact with your account. Be sure to enable SSL when using the mongoimport and mongorestore tools. THROUGHPUT The duration of your data migration depends on the amount of throughput you set up for your collections. Increase the target container s throughput for larger data migrations. The most commonly recommended throughput value for migrations is 10,000 RU/s. MongoDB After you've completed the migration, decrease the container s throughput to save costs.

40 P R E - M I G R AT I O N - C O L L E C T I O N S PRE-CREATE COLLECTIONS Collections that are created using defaults (either via an import tool or an SDK) are provisioned as: Fixed-size collection (no partition key) 1,000 RU/s 1 Before you start your migration, pre-create collections and specify the appropriate settings for each collection. CONFIGURE PARTITION KEY If you need more than 10GB worth of storage, you must configure a partition key as part of provisioning an unlimited container.

41 T O O L I N G - C A S S A N D R A CQLSH Installed as part of Apache Cassandra Use the CQL COPY command to copy local data to Azure Cosmos DB using the Cassandra API endpoint COPY exampleks.tablename FROM filefolderx/*.csv SPARK You can optionally create a Spark cluster to perform a one-time or regularly-scheduled ingestion job to import data into Azure Cosmos DB using the Cassandra API endpoint.

42 T O O L I N G - M O N G O MONGOEXPORT/MONGOIMPORT Translates data from BSON to JSON Higher degree of type compatibility because of JSON data type MongoDB MONGODUMP/MONGORESTORE For larger databases, faster than export/import No translation of data types Potential feature mismatch Could happen if you jump from a much older version of Mongo

43 T O O L I N G - M O N G O MONGOEXPORT/MONGOIMPORT mongoimport.exe --host <your_hostname>: u <your_username> -p <your_password> --db <your_database> --collection <your_collection> -- ssl --sslallowinvalidcertificates --type json -- file C:\sample.json MongoDB MONGODUMP/MONGORESTORE mongorestore.exe --host <your_hostname>: u <your_username> -p <your_password> --db <your_database> --collection <your_collection> -- ssl --sslallowinvalidcertificates <path_to_backup>

44 T O O L I N G D ATA M I G R AT I O N T O O L DATA MIGRATION TOOL The Data Migration tool is an open source solution that imports data to Azure Cosmos DB from a variety of sources, including: JSON files MongoDB MongoDB SQL Server CSV files Azure Table storage Amazon DynamoDB HBase Azure Cosmos DB collections MIGRATING FROM MONGODB You can use MongoDB as a source to migrate data into a Azure Cosmos DB container that s using the SQL API.

45 W H Y A Z U R E C O S M O S D B F O R N O S Q L A P P S? Fully-managed database services, on the world s most trusted cloud AZURE COSMOS DB PROVIDES + FULL DATABASE MANAGEMENT Global distribution Elastic scale-out Guaranteed millisecond latency Choice of consistency models Enterprise-level SLAs Multi-model and multi-api Leverage existing MongoDB drivers, libraries, and tools Turnkey geo-replication Automatic indexing Automatic partitioning Enterprise-grade security Money-back guarantees on high availability and latency Native integrations for serverless architecture, advanced analytics, and event processing Industry-leading TCO (total cost of ownership) vs on-prem or cloud

46 M I G R AT E M O N G O D B A P P S T O A Z U R E C O S M O S DB DISCOVER MIGRATE OPTIMIZE Identify the best migration approach for your apps Microsoft Data Migration Guide Azure Cosmos DB TCO (Total Cost of Ownership) Make the move Import MongoDB data Migrate an existing Node.js MongoDB web app Unlock your apps potential Build a MongoDB API web app with.net and the Azure portal Build a MongoDB API console app with Java and the Azure portal

47 M I G R AT E C A S S A N D R A A P P S T O A Z U R E C O S M O S DB Announced in November, Azure Cosmos DB API for Apache Cassandra is now in preview. Request preview invitation. DISCOVER MIGRATE Identify the best migration approach for your apps Microsoft Data Migration Guide Azure Cosmos DB TCO (Total Cost of Ownership) Make the move Quick start guide: Build a Cassandra app using.net Quick start guide: Build a Cassandra app with Java Quick start guide: Build a Cassandra app with Node.js Quick start guide: Build a Cassandra app using Python Import Cassandra data to Azure Cosmos DB Quick start video

48 H O W C A N I G E T A D D I T I O N A L H E L P W I T H A Z U R E C O S M O S D B? MSDN StackOverflow Requesting New Features? Need Technical Support? Other Questions? UserVoice Submit a Support Ticket in the Azure Portal askcosmosdb@microsoft.com

Azure Cosmos DB Technical Deep Dive

Azure Cosmos DB Technical Deep Dive Andre Essing Technology Solutions Professional Microsoft Deutschland GmbH Andre advises customers in topics all around the Microsoft Data Platform. Since version 7.0,