Cloud Data Management System (CDMS)

Transcription

1 Cloud Management System (CMS) Wiqar Chaudry Solu9ons Engineer Senior Advisor

2 CMS Overview he OpenStack cloud data management system features a canonical data modeling framework designed to broker context sensi9ve data to distributed applica9ons. Key features include: unable consistency and availability guarantees based on transac9ons types emand based replica9on of canonical data to various data management systems (Cassandra, Hadoop, Mongo, MySQL, etc ) ynamic scalability for cloud- scale applica9ons

3 CMS - Components Network level resource manager monitors all physical and virtual compute resources par9cipa9ng in a CMS. his component is responsible for federa9ng all requests to and from CMS Agents CMS manager Compute level agents manage all data and logic processing on physical or virtual hosts. CMS agent CMS agent Supported databases and data management systems. Columnar (Read op9mized) Rela9onal (Write op9mized) ocument (ynamically structured) Canonical data and persistence maintain a golden copy of all data in a consistent state. Canonical data Canonical persistence (Block storage) <Variable>

4 CMS: Fundamental Building Blocks Connec<ons Schemas Mappings Canonical Schemas A?ributes Logic Rule Workflow Rule Rule A logically grouped collec9on of the above metadata objects represents a canonical mapping object. he persisted state of these objects enables sta9c and dynamic analysis of the CMS environment. Func9onal metadata enables flexibility and reusability. Page 4

5 Metadata: Sources and argets Connec<ons Schemas Connec9ons are reusable ar9facts that capture informa9on required to connect to a source or target. Users are able to store this connec9on informa9on and reuse it when extrac9ng or loading data. Schemas define the physical layout, format, and data types of data within a source/target object. Schema ar9facts are also stored and can be reused. Page 5

6 Metadata: Canonical A?ribute Ar<facts Schema Mapping Sets Schema A?ributes Schema Schema mapping sets define the rela9onship between physical source schemas to internal canonical data objects. Schema mapping sets can have a one- to- many rela9onship between physical and logical schemas. A logical data table is a collec9on of one or more a]ributes that defines a data table within the CMS. An a]ribute schema is the collec9on of metadata required to define a managed a]ribute for use within a data table schema. *A?ributes for all intents and purposes are simple key value pairs. Page 6

7 Metadata: Applica<on Logic Logic Workflow Rule A Logic Rule is a reusable object that contains transforma9on logic. A Workflow Rule contains logic that replicates, moves, or makes data available within a requested context. A Rule is a reusable object that contains both data and workflow rules. Page 7

8 A]ribute Schema Associa9ons Created and managed by the system. *,* A?ribute Schema System A?ribute isplay CMS ype User defined name that uniquely iden9fies an a]ribute within a folder. User defined name that uniquely iden9fies an a]ribute within data table. Compound ype A collec9on of a]ributes and CMS ypes that defines a complete or par9al record as a single compound type. CMS ype ype Field FormaPng ( , SSN, phone) Constraints (min/max or allowed values) Primi<ve ype A collec9on of valida9on criteria that can be applied to a]ributes as a template.

9 Rule Associa9ons Rules Canonical External Rule Parameter O- CAP ype Rule Logic Rule Associa<on Parameter isplay /List Correla<on CMS ype Constraint ype A?ributes O- CAP ype isplay System List Input Output

10 CMS Canonical Sources and targets Master catalog of all metadata objects Canonical data foundational structures op level objects (accounts, customers, etc ) Files bases Cloud Applica9ons Canonical Map Object Customer Customer Customer Object transac9ons ransac9on ransac9on ransac9on Object transac9on ransac9on etail ransac9on details etail ransac9on etail Object references ic9onary ic9onary ic9onary Object transactions (aggregate summaries) Object details (logs) Miscellaneous reference and relationship data *All objects contain at least one or more a?ributes

11 Canonical Object Map Source Schema A]ributes (key value pairs) A]ribute Associa9ons bases MySQL Cassandra Etc Applica<on Files Web forms Etc fname lname sms first last mobile CC/MM Logic System First Last Home Work SMS Constraint Logic O- CAP ype SMS Valida9on Logic isplay I First Last I First Last s Reject Records I bases Applica9ons

12 Canonical Model K Key: auto- generated, managed by system, uniquely iden9fies a logical data model. R O R O Object: a logical collec9on of one or more a]ributes. O R K O O R ransac9on Object : manages aggrega9ons and summaries of one or more objects. ransac9on etails: Manages details that might roll up into a transac9on object. R Reference Object: Manages logical references and rela9onship data between the other object types within the system.

13 Why CMS? Single canonical representa9on of data across public and private could environments Context sensi9ve bi- direc9onal replica9on of data Object and collec9on level consistency tuning. Enables collabora9ve data management strategies across enterprises High availability an elas9c scalability.