Become a Champion Data Modeler with SQL Developer Data Modeler 3.0

Transcription

1 Become a Champion Data Modeler with SQL Developer Data Modeler 3.0 Marc de Oliveira, Simplify Systems Introduction This presentation will show you how I think good data models are made, and how SQL Developer Data Modeler 3.0 can help you succeed even large data modeling tasks with hundreds of business terms that need to be defined and structured. The presentation will include data modeling theory as well as specific tool features such as diagram features, model metadata, the Transformation function, and the Reporting Schema export tool. 1

2 ER-modeling or Object Modeling Right off the bat we have to decide on a modeling technique to use in this description of good data modeling. ER and OO are not just two different names for more or less the same thing. Of course you can do data modeling using both techniques but the following two main reasons make me choose Entity Relationship Modeling before Object Oriented Modeling: Relationships: A key difference between ER and OO is how they perceive the things of interest. While OO sees objects as complete things, ER sees entities as elementary things. An ER modeler has to assemble what corresponds to an object, from numerous related entities. I think that this tendency to decompose objects helps the business to see how its information can be used in new and unconventional ways. Syntax: Finally, I find that the Barker Notation of ER is easier to read and communicate than the more modern UML syntax. The main reason for this is that the Barker Notation was developed for conceptual data modeling, while UML is primarily for physical data modeling, even though David Hay has suggested a way to best use UML for conceptual modeling. Of course, this topic of ER vs OO alone can fill an entire paper, so let us just stop with this choice of using the Barker Notation in this article, and get on with a discussion of how to do good data modeling. Business Focus To me, the key to good data modeling is business focus. Data modeling is not primarily about data normalization, actually, it is not even about creating tables for a database. Normalization is about fixing data and relationships in an existing database. It may very well result in a well structured database but normalization techniques will not help you define a database structure that actually fits your business. That is where data modeling comes in. When data modeling is done right it will help you understand and describe a good information structure that fits your business. The normalized database tables are merely bi-products of modeling your business's information requirements. If you do a good job of modeling your business's requirements, you are almost guaranteed to get a good database in return. So, step one in creating a good data model is to concentrate on understanding the things of interest to the business: What are they? And what is their meaning to the business? Naming entities By some, entities are named in plural (such as EMPLOYEES or DEPARTMENTS), while others use names in singular (such as EMPLOYEE or DEPARTMENT). I think there is an important value in naming entities in singular as it draws more attention to the meaning behind the term, while the plural version brings more focus to the rows of data that are going to end up in the database tables. 2

3 During data modeling it is more important to focus on what a term, such as ENROLLMENT, means to the business than how actual course or travel enrollments may end up being stored in the database. In other words, the singular form indicates that your are describing the entity types instead of the actual entity records. It is very important to define good entity names, even though a database developer may think that one name may be just as good as another, as long as the correct data get stored in the correct database tables. The entity names are not just used for defining table names but also (and more importantly) to define business terms, and business terms have a much wider audience than table names. A business term should be understood by (or at least understandable for) everybody in the business. The entities that you define should correspond to the terms that people in your business would use about the same thing. Often business terms are not very clearly defined, which may result in situations where people may use different terms about the same thing or even the same term about different things. In those situation a business analyst can use data models to define a single consistent terminology list that covers all things of interest to the business. That is the focus you should have when defining your entity names. Naming attributes Actually, attribute names should be considered much in the same way as entity names. They, too, represent business terms that should make sense to the people of the business in general and not just IT people that will use the resulting database columns in their application systems. If ENROLLMENT is a business term that require a definition then ENROLLMENT IS ACCEPTED is also a business term that should be explained even though it will be implemented as an attribute instead of an entity. I have found it useful to always name my attributes with an expression that begins with the name of the owning entity. In the previous example IS ACCEPTED would be the attribute name of an attribute of the entity ENROLLMENT. This principle will provide a consistent naming strategy and allow you to express the attributes as stand-alone business terms, such as ENROLLMENT IS ACCEPTED, ENROLL- MENT TIME, and ENROLLMENT COMMENT that can be included in a business terminology list. Describing entities and attributes The entity and attribute description is a textual definition of the entity/attribute, primarily for the purpose of explaining what the business term that it represents, means. First it will help the business analyst/system analyst understand the business that she is analyzing. Second it will serve as a definition for the rest of the business to use and reference in different situations of their work process. Finally, the definitions can serve as context sensitive help text on the application systems built on the resulting database. 3

4 There is no doubt that a business terminology list has immense value for a business in itself. When you add to it that the list is synchronized with easy to read visual diagrams (I will come back to how your diagrams will become easy to read) as well as the help text of the business applications, then it justifies quite a bit of work getting the best possible wording defined for every single business term. Keeping these definitions synchronized like this can be done relatively easily in SQL Developer Data Modeler (SSDM), using the SDDM Reporting Schema. I have written a separate article about the Reporting Schema ( SQL Developer Data Modeler 3.0 Under The Covers: The Reporting Schema ) as well as an article about how to synchronize SDDM data model definitions with on-line help text in APEX ( SQL Developer Data Modeler / APEX Integration Architectural Software Development ). You can find these articles on my web site: As for the structure of your entity and attribute descriptions, I suggest that you always phrase them in a way that could replace the term itself. As an example, the description of the term ENROLLMENT could be something like: participation of someone in one of our company's events, such as a course or conference. Again this will give a consistent structure to your definitions and make them easier to reuse in a natural way in different dynamic contexts, such as business terminology lists or on-line help text. Business Perspectives Another very important aspect of good data modeling is to be aware of the different perspectives of the people that will use your data models. Do not lump all your work together as a single delivery but understand the background of your different recipients. This strategy will emphasize the usefulness of your data modeling work in all areas of the business. I have found that the Zachman Framework provides a very good differentiation between business recipients. If you do not know about the Zachman Framework, I have written an introduction to it called What is the point with the Zachman framework? that you can find on my web site. Basically, the Zachman Framework divides system information actors into the following categories: PLANNERS: They only need the big picture of your entire data model, ie the main information areas that your complete data model covers. They want to be able to understand your entire model of all business areas from a single diagram or list with only a few elements. BUSINESS OWNERS: They need a more detailed conceptual model than the planners but they do not need to understand all business areas. They need to be able to view their own business area without too much interfering elements from other business areas. ARCHITECHTS: They need a complete and consistent conceptual model that describe all business areas including all the relationships between business 4

5 areas. DESIGNERS: They need a complete technical model describing the structure of the database including tables, columns, foreign keys, etc. BUILDERS: They need detailed specifications of all database elements to be able to code application systems. These five perspectives are all important to the business but designers and builders will not be using the high level descriptions very much, while planners and business owners will hate to be confronted with the detailed artifacts. So data models have to be provided on all five levels, and of course they have to be synchronized. Except for a few details that I will address SDDM can support most of your data modeling requirements on all these levels. Besides creating documentation on all five levels it is important to keep the documentation levels separate so as not to confuse people with documentation that was not intended for them. Data Models for Planners Planners do not really need a data model diagram but they still need an overview of what you are modeling. I have found that what is useful on this level is a general grouping of the entities. Depending on the scope of your project, you may easily end up with hundreds of entities in your data model. With models of this size I find it necessary to identify information areas that the entities can be grouped by. Examples of these information areas could be (depending on your project) PERSONS/ORGANIZATIONS, FI- NANCE, MOTIVATION, INVENTORY, etc. These are not necessarily the key entities but separate general entity areas. Sometimes it could make sense to name them after a key entity, though. You should try to restrict yourself to somewhere around 5-10 information areas to make the data overview communicable on a single diagram. It would be nice to be able to create a Core Diagram in SDDM (as described by Peter Weill and Jeanne Ross in their book Enterprise Architecture as Strategy ), but that is not supported by the tool. As SDDM does not have a diagram syntax for representing things like information areas, I have chosen to use the Entity Boxes of the Logical Diagram to represent them, as shown on Illustration

6 Illustration 1: Information Areas As you can see, I have also assigned a different color to each information area. I reuse the same colors in my ER Diagrams to signal what kind of entities the diagrams contain. I will say more about that in the next section. To avoid that these information areas are converted into tables by the Engineer to relational function of SDDM, you can un-check the Engineer to check-box (unfortunately, a bug in SDDM 3.0 ignores this but, eventually, this should prevent these information area entities to be converted to tables). Alternatively, you could decide to create an Information Area Diagram or even a real Core Diagram in another diagramming tool. As the Information Area Diagram and Core Diagram do not have too many elements, it is not so much work to keep your information areas synchronized in another tool. The Information Area Diagram of Illustration 2, and the Core Diagram of Illustration 3 has been created in Power Point. 6

7 Illustration 2: Information Area Diagram Illustration 3: Core Diagram These are good diagrams for communicating the overall structure of an entire appli- 7

8 cation system. Data Models for Business Owners Business Owners may like the Core Diagram or information area diagram described in the previous section as an overview but when they need to discuss their use of data within their business area, they will need real ER Diagrams with focus on specific entity categories. An entity category is a categorization of entities within an information area. The idea is to divide the entities into small chunks that can be described and understood on a single ER Diagram. As an example the information area PERSON/ORGANIZATION could be divided into the entity categories PARTY, AGREEMENT, and INTEREST. The relationship between information areas, entity categories, and entities is hierarchical as shown on Illustration 4. Organizing entities like this will allow you to create focused ER Diagrams that are much easier to read than if you just let your ER Diagrams build up from some central entity(s). Illustration 4: Entity Category Usually, each business owner will be interested in a few information areas, and presenting her to the relevant parts of the data model from the top will help add structure to your data model and, hence, make it easier to understand. Business owners should be presented to a list of relevant information areas with related entity categories, as shown on Illustration 5. This will give them an overview of the entire model, while making it easy for them to dive into their specific areas of interest. 8

9 Illustration 5: Entity Category List When they dive into one of the entity categories, they should be presented with a set of ER Diagrams that describe the chosen category. Depending on the complexity of the entity category this is best done with a set of diagrams that build up to an entity category overview. In SDDM ER Diagrams are SubViews of the Logical Model. Begin with the simplest possible ER Diagram of the chosen entity category, maybe as little as a single entity never more than three or four entities. Then add a few entities to the simple diagram to show more and more relationships until you end up with an entity category overview that describes the entire entity category. Always keep the diagrams as simple as possible. Remember that they have to be read by people that are not necessarily used to data modeling. Let me just add a few basic pointers about what I do to make easy-to-read ER Diagrams. Keep to vertical and horizontal relationship lines, they keep the diagrams calm and nice to look at. Avoid crossing lines (if possible). Even when it is obvious how the lines are crossing, it does add unnecessary ambiguity and confusion to the diagram. Avoid bending lines. A bend is a syntactic element even if it has no meaning. To the brain it is one extra unnecessary thing to process. Crows fly south and east. Place your master-detail relationships with the detail end on top or to the left. That will create diagrams where the most interest- 9

10 ing elements are in the top left corner. Ie you will quickly get a sense of what the diagram is about. Try to keep even your largest diagrams under 10 entities. Each diagram should be focused on explaining a single area (entity category), so verify if you are not trying to express too much, if you feel that you have to use more entities on your diagram. David Hay has quite a bit to say about this kind of diagramming principles in his books and articles. I can recommend everything he writes. The following, Illustration 6, could be an example of a simple ER Diagram of the Event Definition entity category. Illustration 6: Event Definition - simple It shows that an event definition (that could be a course variant) is based on a conceptual course, and that academic areas may contain one or more conceptual courses. This alone provides content for good discussions about how event definitions are used by the business, and it also sets the scene for the next diagram (Illustration 7). 10

11 Illustration 7: Event Definition - Course Format This diagram contains more entities but when one has understood the previous diagram this diagram does not seem so overwhelming. This step-by-step method of describing a data model is much easier for people to follow even with no prior knowledge about data modeling. The ER Diagrams may very well include entities from other related entity categories. This is where the entity category colors will help keep the focus on the central entities of the diagram. Consider the following entity category overview diagram (Illustration 8) that shows how Event Definitions are related to entities from the Schedule and Person/Organization information areas (each event definition may be planned as one or more planned events, and every academic area may be managed by some party). 11

12 Illustration 8: Event Definition - Overview If the entity category is very complex, the entity category overview does not have to show all related entities but just enough to give people an overview of the category. Data Models for Architects The architects need diagrams with all the details. As they know how to read ER Diagrams they do not need the step-by-step diagrams (of course, even an architect may wish to look at the business owner's diagrams from time to time) but wish to be sure that they are seeing everything. By keeping the focus of the entity categories we can build very detailed ER Diagrams that are still not too complicated to understand. In some cases the detailed ER Diagrams may be the same as the business owner's entity category overview diagram but often the architect's diagram will contain extra entities and relationships that supports the business without being of direct interest to the business owners. Consider the architects' Detailed ER Diagrams to be the next step after the Entity Category Overview Diagrams. They will usually be too confusing to discuss with a 12

13 business owner that does not have data modeling experience but will be very suited for discussions between architects or with database developers. Data Models for Designers and Builders When it comes to application designers, the models need to be of database tables, columns, and foreign keys, so you will have to convert your nice logical model to a (in SDDM terms) relational model. This is easily done using the Engineer to Relational Model function under the Design menu. The function converts the currently open ER Diagram to a relational model. There is a number of preferences and other settings that helps you control how the engineering function should work, and even if it still does not work exactly how you would want it to work, then take a look at the Transformation function of SDDM (Tools > Design Rules > Transformations). This is a very powerful scripting tool that will allow you to automate updates to both your logical and relational models. SDDM comes with a few useful scripts that can inspire you to build your own. A very useful script that comes with SDDM can add columns from a template table to all tables that do not have them. This can be used to make sure that audit columns are added to all tables. I have extended one of the shipped scripts that converts table and column names to lower case to also replace spaces ( ) and dashes ( - ) with underscore ( _ ) characters, like this: tables = model.gettableset().toarray(); for (var t = 0; t < tables.length; t++) { table = tables[t]; tablename = table.getname().tolowercase(); } if (tablename!= tablename.replace(" ","_").replace("-","_")) { table.setname(tablename.replace(" ","_").replace("-","_")); } columns = table.getelements(); size = table.getelementscollection().size(); for (var i = 0; i < size; i++) { column = columns[i]; cname = column.getname().tolowercase(); cname = cname.replace(" ","_").replace("-","_"); if (column.getname()!= cname) { column.setname(cname); } } Documentation for the Transformation function is not very good but the SDDM developers are very helpful on the Oracle SDDM Forum. You will appreciate that you kept the numbers of entities down on your ER Diagrams as your tables take up much more space on the relational diagrams. This is primarily because data types are shown on the relational model but it is also because your tables will contain more columns than your entities had attributes. First you have all 13

14 the new foreign keys columns and possibly columns from subtype entities, but you may also have audit columns etc that you would not include in your logical model. So the relational version of Illustration 7 above will look like Illustration 9: Still, the relational diagrams are very useful to programmers that need to develop the application. During engineering you will often realize that your names will break the Oracle database's 30 character limit. These can usually be handled by the Name Abbreviation function (Tools > Name Abbreviations). This function allows you to specify a text document with a list of words and their abbreviation. When you run the function all tables, columns, indexes and views can be abbreviated according to the list you have provided. This allows you to use a natural language naming strategy in your logical model and still create a syntactically correct relational database schema. Try to avoid any changes of individual elements in the relational model. You do not wish to have to maintain two models, so make sure that your relational model can always be created by running the Engineer to Relational Model function, the Name Abbreviations function, and maybe one or two Transformation scripts. This will give you the freedom to focus on your logical model, even after your designers have started working with the database. So when a new column needs be created, a new unique identifier is needed, or a foreign key needs to changed, you can just apply the changes to your logical model and rerun the engineering functions and scripts to have the changes reflected in the relational model. This strategy will assure that your logical model is always synchronized with the application even after the application is in production and maintenance is going on. This means that your business owner's diagrams are always updated with the latest changes in the database, because any changes to an entity or a relationship, in the logical model, is automatically reflected in all ER Diagrams that the entity is showed on. Reusing your logical model Many developers forget about the logical model when they start working on the relational model but by keeping your focus on the logical model, you will have the great advantage of being able to keep talking to the people using business terms instead of referring to table names, column names, and foreign key names. And you can reuse the logical model for much more than just maintaining your relational model. SDDM has a function that exports both your logical and your relational model to a database schema, so you can access all your definitions, including your descriptions and notes, from pl/sql. SDDM calls it the Reporting Schema and you can generate it by clicking File > Export > To Reporting Schema. I have written about how you can reuse your logical models through the Reporting Schema in my article SQL Developer Data Modeler 3.0 Under The Covers: The Reporting Schema. I have also written an article about 14

15 how to integrate the Reporting Schema with your APEX applications in my article SQL Developer Data Modeler / APEX Integration Architectural Software Development. Both articles can be found on my web site: Conclusion Data Modeling is a big task if you want to do it right. As I see it it is not worth doing, if you are not going to do it right. You have to invest a lot in it to be able to avoid redoing everything later. The good news are that if you do it right, all the work you put into building your logical model, you will not have to do when creating the relational model. The application will not work on a half baked database, so even if you try to cut corners during the logical modeling because you think it is less important than the relational modeling, you are just wasting time, as you will have to think about all the consequences during the relational modeling, anyway. If you spend that same time up front instead you will win a great logical model spending the same time as if you would only build a relational model. And your business will love you for it, as people will have a much easier time understanding what you are telling about their information needs. And I am sure you will get to understand the business better, too. If you have any comments about this article, you can always contact me through my web site: Good luck! 15

16 16