The Data Organization

Transcription

1 The Data Organization 1251 Yosemite Way Hayward, CA (510) Meta Data By Rainer Schoenrank Data Warehouse Consultant Nov 2010

2 Copyright 2010 Rainer Schoenrank. All rights reserved. No part of this document may be reproduced in whole or in part, in any form or by any means, electronic or manual, without express written consent of the copyright owner. The logo is a trademark of The Data Organization ( in the United States and/or in other countries. Biography Rainer Schoenrank is the senior data warehouse consultant for The Data Organization. He has degrees in physics from the University of Victoria and computer science from the University of Victoria and California State University Hayward. He has built data warehouses for clients such as Pacific Bell, Genentech, GE Leasing, SGI, PPFA, Brobeck, BofA, Clorox and Leapfrog. He can be reached at rschoenrank@computer.org. 3/5/2011 Copyright 2010 Rainer Schoenrank Page 2

3 1. INTRODUCTION There is no standard for meta data and there never will be because both front end and back end tool vendors create meta data without regard for standards or centralized administration. The best that a data warehouse team can do is to carefully document all the forms of meta data (Kimball) Literally, meta data is data about data. So what is data and what do we need to know about our data? The Oxford English Dictionary defines data as something known or assumed as fact and made the basis of reasoning or calculation. Both of these definitions are too simplistic. They don t help us understand what data are, they don t describe the properties of data and they don t help us reason about data. To get a better understanding of meta data, let s look at data without meta data: DATA value 100, North Fifth West The data looks nice but what does it mean? The meaning of the data is given by its meta data. 3/5/2011 Copyright 2010 Rainer Schoenrank Page 3

4 2. WHAT IS META DATA So, let us look deeper and ask how are data created and to what purpose. People create data by recording symbols that represent facts. The reason people create data is to describe their world and the events that happened. Data is the description of the physical world written in a language/dialect/jargon. You are reading data because I have recorded symbols to describe my understanding of data. The symbols are not the letters on the page, but the words (groups of letters). The groups of letters are the representation of the symbols. The representation could as easily be pictograms, glyphs, etc. To understand the symbols, you must achieve dual representation: you must mentally represent the symbol as well as the relation between the symbol and what it stands for. The symbols are organized into sentences and the sentences are organized as paragraphs according the rules of the grammar of the language. You understand these symbols because they have been organized into a language and we are all familiar with the meta data of languages (for example, English). The meta data of a language has four components: A method of creating and representing symbols (an alphabet) an inventory of the symbols and their meaning (a dictionary) rules for organizing symbols into groups (a grammar) relationships between symbols (a thesaurus) that list: o the same symbol with different meanings (homonyms) o different symbols that mean the same thing (synonyms) 3/5/2011 Copyright 2010 Rainer Schoenrank Page 4

5 3. THE IMPORTANCE OF META DATA At the beginning, we saw data without any meta data. Here is the same data with its meta data. DATA value 100, North Fifth West META DATA entity name patient patient patient attribute name weight height street address description the patient's measured weight the patient's measured height the patient's address data type measured quantity measured quantity address display format european number decimal number string The meta data allows us to give meaning to the data and interpret the data with the statement: the patient is too short for the weight 3/5/2011 Copyright 2010 Rainer Schoenrank Page 5

6 4. META DATA FOR BUSINESS USERS More usefully, in a business environment, meta data is documentation and information about the data structures such as files, databases, data marts, and decision support models that support the business in keeping its records and making its decisions. The documentation describes what kind of data is stored, where it is located and how it is related to other data. In a data warehouse environment, the data warehouse is the archive of all of the transactions completed and the meta data of interest is documentation about the operational source of the data, how it is mapped onto the data warehouse and the warehouse is mapped onto the decision support models. In summary, business meta data includes: Inventory of data structures o logical data structures o physical data architecture and storage specifications, how it is encoded o semantic (consumer-friendly) descriptions of data o business definitions of data entities Additionally, businesses also require: Inventory of the processes that create and use the data Relationship between data structures and processes o data source and state transition information o where it comes from, o how was it calculated o how it is related to your business 3/5/2011 Copyright 2010 Rainer Schoenrank Page 6

7 5. ISSUES The collection of meta data must be an automated function of the meta data repository so that the meta data is always current. If meta data collection is not automated, then it becomes one of the application development tasks and revisions to the meta data will not be current. Business users assume that a business concept exists only once in the meta data repository. Actually, there are many columns/fields/attributes that are used by different processes to record the same business concept. This raises a number of issues. There's more than one version of meta data for a data item. Reasonable people can disagree forever on how to describe something. Requiring everyone to use the same vocabulary to describe their material denudes the cognitive landscape and enforces homogeneity in ideas. People don t accurately record meta data. People are lousy observers of their own behaviors. Entire religions are formed with the goal of helping people understand themselves better; therapists rake in billions working for this very end. Why should we believe that using meta data will help J. Random User get in touch with her Buddha nature Different users create meta data for different purposes. Meta data exists in a competitive world. When a data organization or classification will confer an advantage, people will arrange their meta data to use that advantage. People steadfastly refuse to collect meta data even when there's a positive benefit to creating good meta data. 6. REFERENCES Judy S. DeLoache, Mindful of Symbols, SciAm, August 2005, pp /5/2011 Copyright 2010 Rainer Schoenrank Page 7