What Multisite Means for Identity Management

Transcription

1 What Multisite Means for Identity Management Multisite Howard Huang, Huawei

2 2 Outline Multisite Project Overview Multisite Identity Management Use Case Proposed Solution Demo Upstream Bug Report

3 3 Multisite Project Overview Multisite project was approved on Apr 2015, to be a requirement project that deals with OPNFV deployment across multiple sites Weekly meeting at #opnfv-meeting very Thursday, have members from various companies to participate (Ericsson, Orange, Docomo, Metaswitch, RedHat ) Discussed 5 use cases: - Multisite Identity Service Management (Use Case 1) - Multisite VNF HA Support (Use Case 2) - Multisite VNF Geo Redundancy Support (Use Case 3) - Multisite Centralized Services (Use Case 4&5) Feedback BPs or bugfixes back to the upstream community based on the discussion:

4 4 Multisite Identity Service Management - Overview Use Case Description: User should be able to manage virtual resources spread over multiple OpenStack regions, by using a single authentication point

5 5 Multisite Identity Service Management Requirement Analysis A user is provided with a single authentication URL to the Identity (Keystone) service. Using that URL, the user authenticates with Keystone by requesting a token typically using username/password credentials. The keystone server validates the credentials, possibly with an external LDAP/AD server and returns a token to the user. With token type UUID/Fernet, the user request the service catalog. With PKI tokens the service catalog is included in the token. The user sends a request to a service in a selected region including the token. Now the service in the region, say Nova needs to validate the token. Nova uses its configured keystone endpoint and admin (?) credentials to request token validation from Keystone. The Keystone token validation should preferably be done in the same region as Nova itself. Now Keystone has to validate the token that also (always?) includes a project ID in order to make sure the user is authorized to use Nova. The project ID is stored in the assignment backend - tables in the Keystone SQL database. For this project ID validation the assignment backend database needs to have the same content as the keystone who issued the token. So either 1) services in all regions are configured with a central keystone endpoint through which all token validations will happen. or 2) the Keystone assignment backend database is replicated and thus available to Keystone instances locally in each region. 2) is obviously the only scalable solution that produce no inter region traffic for normal service usage. Only when data in the assignment backend is changed, replication traffic will be sent between regions. Assignment data includes domains, projects, roles and role assignments.

6 6 Multisite Identity Service Management Key Tech Points Keystone Deployment: Centralized: a single Keystone service installed in some location, either in a "master" region or totally external as a service to OpenStack regions. Distributed: a Keystone service is deployed in each region Token types: UUID tokens,pki tokens, Fernet tokens, Cryptographic tokens Database replication: Master/slave asynchronous, Multi master synchronous, Symmetrical/asymmetrical Database server sharing: In an OpenStack controller normally many databases from different services are provided from the same database server instance. For HA reasons, the database server is usually synchronously replicated to a few other nodes (controllers) to form a cluster. Note that _all_ database are replicated in this case, for example when Galera sync repl is used. Only the Keystone database can be replicated to other sites. Replicating databases for other services will cause those services to get of out sync and malfunction. Since only the Keystone database is to be sync replicated to another region/site, it implies that the Keystone database is deployed into its own database server. This affects existing installers that would need to be more flexible. If the database server can be shared when async master/slave repl is used, should be investigated. (Most likely not due to the need to enable global transaction identifiers GTID)

7 7 Multisite Identity Service Management Preferred Solution KeyStone service(distributed) with Fernet token + Async replication ( star-mode). One master KeyStone cluster with Fernet token in two sites (for site level high availability purpose), other sites will be installed with at least 2 slave nodes where the node is configured with DB async replication from the master cluster members, and one slave s mater node in site1, another slave s master node in site 2. Only the master cluster nodes are allowed to write, other slave nodes waiting for replication from the master cluster ( very little delay) member. But the chanllenge of key distribution and rotation for Fernet token should be settled, you can refer to these two blogs: Pros. Why cluster in the master sites? There are lots of master nodes in the cluster, in order to provide more slaves could be done async. replication in parallel. Why two sites for the master cluster? to provide higher reliability (site level) for writing request. Why using multi-slaves in other sites. Slave has no knowledge of other slaves, so easy to manage multi-slaves in one site than a cluster, and multi-slaves work independently but provide multi-instance redundancy(like a cluster, but independent). Cons. The distribution/rotation of key management.

8 8 Multisite Identity Service Management Site 1 Site 2 Nova API Nova Local Token Validation Local Token Validation KeyStone KeyStone DB Cluster DB Cluster DB Cluster DB DB DB DB DB DB DB DB DB KeyStone KeyStone KeyStone Local Token Validation Local Token Validation Local Token Validation Nova Nova Nova Site 3 Site 4 Site x

9 9 Multisite Identity Service Management Now there is something different: We demoed the solution to see if it is logical! (Shout out thanks to Hans Feldt from Ericsson)

10 10 Multisite Identity Service Management Demo Intro: Instructions and tools to start a mini multi region cloud on a single host using docker containers. The only available services are keystone and glance, working to some extent. The (current) goal is just to be able prototype multi region cloud. To follow these steps you probably need a fairly recent docker version like 1.7. Short description: Human users are in a singleton (shared by all regions) LDAP server. OpenStack Kilo is used since containers use Ubuntu as base and its OS packages. Keystone image use git repo master branch to get latest fixes A mysql-5.6 database server is started per region Keystone configured to use Fernet tokens mysql master/slave replication between the master DB server and slave DB server(s) Some reuse from the OpenStack Kolla project

11 11 Multisite Identity Service Management BUG: Can't specify identity endpoint for token validation among several keystone servers in keystonemiddleware: The method we provide works. A similar bugfix was merged into Liberty release.

12 Copyright 2015 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.