Normalized Relational Database Implementation of VSAM Indexed Files

Transcription

1 Normalized Relational Database Implementation of VSAM Indexed Files Note: this discussion applies to Microsoft SQL Server, Oracle Database and IBM DB2 LUW. Impediments to a Normalized VSAM Emulation Database The organization of a relational database with its rows and columns presents a particular challenge to the emulation of VSAM Key Sequential Datasets (KSDS). Users of KSDSes (VSAM files) are accustomed to accessing those datasets in a variety of ways that are not compatible with relational databases. For example, START, READ NEXT, READ PRIOR. In addition, the COBOL records that are used to define VSAM files allow OCCURS, REDEFINES and group items, which are not supported in relational databases. Moreover, as a matter of usage, many COBOL developers use VSAM files in ways that are difficult to replicate in a relational database. For example, it is common practice to have a file layout that consists of a fixed part that is common to every record in the file as well as a part that may be different depending on the subtype of the data being stored. By way of illustration consider an Insurance Policy file that contains common information about the policyholder but may have different information depending on whether the policy is a whole life, term, auto or homeowner s policy. These are usually represented in the COBOL record as group items that redefine one another with a field in the fixed portion of the recording indicating what variable format type to use. Another common practice is to have a large master record that contains sections that represent distinct parts of the record. Continuing the insurance illustration, a policy record might have a section with information about the tax exempt status of the policy, coverage values, policy values, prior year coverage, etc. These are usually represented in the COBOL records as group items that do not redefine each other. Solutions to Normalization Impediments In order to address these problems, Language Portability Solutions (LPS) has developed a comprehensive set of normalization tools and techniques that provide flexible and customizable solutions to these impediments. These solutions work with VSAM KSDSes, RRDSes and ESDSes. The remainder of this document describes and illustrates these solutions using VSAM KSDSes. Before proceeding, it is worth nothing that there is a performance tradeoff in normalizing a database that emulates VSAM files. Usually, one of the prime motivators for normalizing the emulation database is to allow future development to use SQL

2 statements to access the database directly as opposed to relying on legacy record accessing and updating techniques to allow access to the relational database using the same methods previously described (START, READ NEXT, etc.). The way in which the COBOL records that describe VSAM files are constructed (see above) often results in records with hundreds, if not thousands, of fields. One VSAM file LPS migrated was a 15,000 character record that contained 373 elementary items, including 18 occurring groups and 4 large redefines. In order to read this record using a normalized implementation, it would be necessary to select more than a thousand columns, transform the SQL data in each column and pack it into a byte array so that it could be used by the migrated COBOL program. To write a record, the process needs to be reversed. This design seriously slows down reading and updating of large files. To address this performance issue, LPS has also developed a high performance implementation (please see our High Performance Relational Implementation of VSAM Indexed Files document) in addition to our normalized implementation. Normalization Strategy LPS s normalization strategy provides solutions to all of the common issues previously mentioned. At its most basic level, a VSAM file record is converted to a single table that contains one column for each elementary item within the record. It is also possible to map group items to columns and rename columns as well as to create Boolean and DateTime database types and map them to COBOL fields. One or more indexes are created on the table to correspond to the primary key and any alternate keys associated with the original VSAM file. If the record is flat and contains no occurs, variable format-style redefines or section-style divisions, then the normalization is considered complete. In the case where any or all of these three impediments to normalization are present, then additional capabilities are present to provide solutions. Normalization of Occurring Items Since relational databases do not support the concept of an array of columns, an alternative method for normalizing these items must be employed. Thus, a COBOL item defined as: MONTHLY-PMTS PIC S9(6)V99 OCCURS 12. could not be supported directly in a relational database. In order to accommodate such items, LPS s VSAM emulation technology provides two methods for normalization: inline occurs or occurs tables. Inline Occurs

3 In order to normalize an occurring item using inline occurs, the database must contain one unique column for each occurring element. Note that this works best when the number of occurrences is small. Returning to the above COBOL item definition: MONTHLY-PMTS PIC S9(6)V99 OCCURS 12. in order to normalize this item inline, the database must be defined with one unique column for each occurring element. Thus the above COBOL item would be declared as follows: MONTHLY_PMTS_1 Decimal(8,2) MONTHLY_PMTS_2 Decimal(8,2) MONTHLY_PMTS_3 Decimal(8,2) MONTHLY_PMTS_12 Decimal(8,2) When a record is read, the VSAM emulation methods handle placing the unique columns into the corresponding occurrence in the migrated COBOL record. When a record is written, each occurrence in the COBOL record is placed in its corresponding unique column in the table. This is handled automatically by the interface methods and is transparent to the application program. Occurs Tables The other normalization option is to have the occurring item placed in its own table, which will be automatically linked to the table that contains the occurring item via its primary key plus an index that indicates the occurrence number. This is often preferable when there are a large number of occurrences or there are many elementary items in an occurring group. For example, a COBOL item defined as: MONTHLY-BILL-HISTORY OCCURS 120. MONTHLY-BILL-AMT PIC S9(6)V99. MONTHLY_PMT PIC S9(6)V99. that occurs within a VSAM file with customer number as its primary key will result in the creation of an occurs table with a primary key of the customer number and an index that represents the occurrence (1-120) and two columns corresponding to the two elementary items. If the occurring item is an elementary item, the table will have the same primary key and one column corresponding to the elementary item. In either case values are only stored in the table if they are not empty or if they have changed. Empty elements or elements that have not changed are not written or rewritten. The

4 accessing of the occurs table is done automatically by the VSAM emulation methods and is transparent to the application program, which continues to process the occurring item exactly as it did originally. Normalization of Variable Formats Variable formats are always normalized by creating a table for each variable format type contained in the original COBOL record. The primary key of these variable format tables is the primary key of the base table. Since each table represents one variable format type, all records of that type are grouped together in the same table. Returning to the insurance policy example, suppose the outline of the policy record looked as follows: 01 POLICY-RECORD. 05 POLICY-NUMBER PIC 9(8). 05 POLICY-ISSUE-DT PIC 9(6). 05 POLICY-TYPE PIC X. 05 POICY-HOLDER-DATA. 05 WHOLE-LIFE-POLICY. 10 WHOLE-LIFE-AMOUNT PIC S9(9)V99 COMP. 10 WHOLE-LIFE-INTEREST-RATE PIC S99v99 COMP. 05 TERM-LIFE-POLICY REDEFINES WHOLE LIFE POLICY. 10 TERM-LIFE-AMOUNT PIC S9(9)V99 COMP. 10 TERM-LIFE-TERM-YRS PIC AUTO-POLICY REDEFINES WHOLE-LIFE-POLICY. 10 AUTO-COVERAGES. 15 AUTO-LIABILITY PIC S9(6)V99 COMP. 15 AUTO-COMP PIC S9(6)V99 COMP. 05 HOME-POLICY REDEFINES WHOLE-LIFE-POLICY. 10 DWELLING-COVERAGES. 15 DWELLING-LOSS PIC S9(9)V99 COMP. 15 CONTENTS-LOSS PIC S9(9)V99 COMP. This record would result in the creation of a base table with the items in the POLICY- RECORD that are common to all policy types and four variable format tables one for each of the group items WHOLE-LIFE-POLICY, TERM-LIFE-POLICY, AUTO-POLICY and HOME-POLICY. Since these items redefine each other, only one variable format table will be written for each policy record written to the base table. Each of these tables will have columns that correspond to the primary key of the base table and a column for each item in the group that makes up the variable format. The decision as to which variable format table will be written is made based on information provided about the field in the fixed part of the record that contains the variable format type (in this

5 example POLICY-TYPE) and the values that determine the selection, e.g. W, T, A or H. LPS s VSAM emulation methods use this information to automatically select the appropriate variable format table when reading, adding, rewriting or deleting records. Since any of the items within the group that defines a variable format type can have occurring items, these items are linked to the variable format table and handled as described previously (see Normalization of Occurring Items ). Normalization of Sections Sections, as previously discussed, are group items that describe major parts of a record that form integral, but different, parts of the record. Each section is required for the record to be complete but they represent different aspects of the record data. Consider an insurance policy record: 01 POLICY-MASTER. 05 COVERAGE-ID PIC XX. 05 POLICY-ID PIC X(10). 05 POLICY-INFO. 10 BASE-COVERAGE-NO PIC S9(4) COMP. 10 POLICY-EFF-DATE PIC X(10). 05 COVERAGE-INFO. 10 COVERAGE-NO PIC COVERAGE-STATUS PIC XXX. 05 PREV-COVERAGE-INFO. 10 PREV-COVERAGE-NO PIC PREV-COVERAGE-TYPE PIC X(8). Unlike variable format types, sections are required for each record in the VSAM file and do not redefine each other. LPS s normalization facility allows these sections to be stored in separate tables that are linked to the base table via its primary key. LPS s VSAM emulation methods automatically maintain these tables when reading, adding, rewriting or deleting records. To provide maximum flexibility, sections may include occurring items as well as variable formats. These are handled as previously described (see Normalization of Occurring Items and Normalization of Variable Formats ). Preparation of the VSAM Emulation Database In order to prepare the VSAM emulation database, several steps are necessary. LPS initially performs these steps as part of the migration process. We also provide clients with a tool to assist them in ongoing maintenance and enhancements to the underlying database once the migration is completed. The initial migration process requires a number of steps depending on whether or not both online and batch programs will be migrated. If online programs are being

6 migrated, then certain FILE section information needs to be extracted from the CICS CSD file so that the VSAM file id used in CICS commands can be related to the correct file. For both batch and online VSAM programs it is necessary to process the JCLs containing the IDCAMS definitions of all VSAM files so that information such as primary and alternate keys, their locations and sizes can be determined. The result of these two steps is the creation of an XML file that contains the basic information necessary to create the VSAM emulation relational database. Once the initial XML file has been created, a utility is used to specify the various normalization options i.e. inline and table occurs, variable formats and sections. The utility uses the XML file as well as the VSAM record definitions during this process. Upon completion, a new XML file is created and a database DDL is generated. Examples Example 1 Video Rental Demo The video rental demo provides a simple example of a VSAM file that contains no variable format parts or occurring items. The Video Master File (VIDMAST) is illustrated below: VSAM COBOL Records 01 VIDEO-MASTER-RECORD. 05 VM-VIDEO-NUMBER. 10 VM-TITLE-NUMBER PIC 9(06). 10 VM-COPY-NUMBER PIC 9(02). 05 VM-TITLE PIC X(30). 05 VM-MEDIA-TYPE PIC X(01). 88 VM-TAPE VALUE 'T'. 88 VM-DVD VALUE 'D'. 88 VM-GAME VALUE 'G'. 05 VM-STATUS PIC X(01). 88 VM-STATUS-IN VALUE '1'. 88 VM-STATUS-OUT VALUE '2'. 05 VM-CUSTOMER-NUMBER PIC X(07). 05 VM-DATE-OUT PIC X(10). XML File <?xml version="1.0"?> <VSAMFiles xmlns:xsi=" xmlns:xsd=" <VSAMRecords /> <Files> <VSAMFile> <TableName>VIDMAST</TableName> <RecordName>VideoMasterRecord</RecordName>

7 <RecordClass>VIDMAST</RecordClass> <KeyInfos> <VSAMKeyInfo> <FileId>VIDMAST</FileId> <DSNName>MSS.PKS.CODE1.VIDMAST</DSNName> <ColName>VIDNUM</ColName> <Length>8</Length> </VSAMKeyInfo> <VSAMKeyInfo> <FileId>VIDMSTA</FileId> <DSNName>MSS.PKS.CODE1.VIDMSTA</DSNName> <ColName>CUSTNUM</ColName> <Offset>40</Offset> <Length>7</Length> <Dupls>true</Dupls> </VSAMKeyInfo> </KeyInfos> <DataCols> <ColName>VIDNUM</ColName> <FldName>VmVideoNumber</FldName> <Size>8</Size> <ColName>VIDTITLE</ColName> <FldName>VmTitle</FldName> <Size>38</Size> <ColName>VIDMEDIATYPE</ColName> <FldName>VmMediaType</FldName> <Size>1</Size> <ColName>VIDSTATUS</ColName> <FldName>VmStatus</FldName> <Size>1</Size> <ColName>CUSTNUM</ColName> <FldName>VmCustomerNumber</FldName> <Size>7</Size> <ColName>VIDDATEOUT</ColName> <FldName>VmDateOut</FldName> <Size>10</Size> </VSAMFile> </Files> </VSAMFiles> Generated T-SQL DDL IF OBJECT_ID (N'VIDMAST', N'U') IS NOT NULL DROP TABLE [VIDMAST]; CREATE TABLE [VIDMAST] ( [VIDNUM] CHAR(8) NOT NULL, CONSTRAINT PK_VIDMAST_VIDNUM PRIMARY KEY (VIDNUM),

8 [CUSTNUM] CHAR(7) NOT NULL, [VIDTITLE] CHAR(38), [VIDMEDIATYPE] CHAR(1), [VIDSTATUS] CHAR(1), [VIDDATEOUT] CHAR(10), LastUpdate DATETIME NOT NULL, LastUser CHAR(15) NOT NULL ) CREATE INDEX IX_VIDMAST_CUSTNUM ON VIDMAST(CUSTNUM) Example 2 Variable Formats and Occurs The following example illustrates a more complex normalization using variable formats and both inline occurs and occurs tables: COBOL Record 01 POLICY-RECORD. 05 POLICY-NUMBER PIC 9(8). 05 POLICY-ISSUE-DT PIC 9(6). 05 POLICY-TYPE PIC X. 05 POLICY-HOLDER-DATA OCCURS POLICY-HOLDER-BILLING PIC S9(6)V99 COMP POLICY-HOLDER-PAYMENT PIC S9(6)V99 COMP WHOLE-LIFE-POLICY. 10 WHOLE-LIFE-AMOUNT PIC S9(9)V99 COMP WHOLE-LIFE-INTEREST-RATE PIC S99V99 COMP WHOLE-LIFE-INTEREST-AMT PIC S9(6)V99 OCCURS TERM-LIFE-POLICY REDEFINES WHOLE-LIFE-POLICY. 10 TERM-LIFE-AMOUNT PIC S9(9)V99 COMP TERM-LIFE-TERM-YRS PIC AUTO-POLICY REDEFINES WHOLE-LIFE-POLICY. 10 AUTO-COVERAGES. 15 AUTO-LIABILITY PIC S9(6)V99 COMP AUTO-COMP PIC S9(6)V99 COMP HOME-POLICY REDEFINES WHOLE-LIFE-POLICY. 10 DWELLING-COVERAGES. 15 DWELLING-LOSS PIC S9(9)V99 COMP CONTENTS-LOSS PIC S9(9)V99 COMP-3. XML File <?xml version="1.0"?> <VSAMFiles xmlns:xsi=" xmlns:xsd=" <Files> <VSAMFile> <TableName>POLICYRECORD_TABLE</TableName> <MaxRecSz>1320</MaxRecSz> <RecordName>PolicyRecord</RecordName> <RecordClass>NORMSAMPLE</RecordClass>

9 <RecordDLL>NormSample.dll</RecordDLL> <VarFormatColumn>POLICYTYPE</VarFormatColumn> <KeyInfos> <VSAMKeyInfo> <FileId>PolicyNumber</FileId> <ColName>POLICYNUMBER</ColName> <Length>8</Length> </VSAMKeyInfo> </KeyInfos> <DataCols> <ColType>Int32</ColType> <ColName>POLICYNUMBER</ColName> <FldName>PolicyNumber</FldName> <Size>8</Size> <ColType>Int32</ColType> <ColName>POLICYISSUEDT</ColName> <FldName>PolicyIssueDt</FldName> <Size>6</Size> <ColName>POLICYTYPE</ColName> <FldName>PolicyType</FldName> <Size>1</Size> <VSAMOccursTable> <Occurs>120</Occurs> <Table> <TableName>POLICYHOLDERDATA</TableName> <MaxRecSz>1320</MaxRecSz> <DataCols> <ColName>POLICYHOLDERBILLING</ColName> <FldName>PolicyHolderBilling</FldName> <Size>8</Size> <ColName>POLICYHOLDERPAYMENT</ColName> <FldName>PolicyHolderPayment</FldName> <Size>8</Size> </Table> </VSAMOccursTable> </OccursTables> <VarFormats> <VSAMVarFormat> <TypeValue>W</TypeValue> <Table> <TableName>WHOLELIFEPOLICY</TableName> <MaxRecSz>1320</MaxRecSz>

10 <DataCols> <ColName>WHOLELIFEAMOUNT</ColName> <FldName>WholeLifeAmount</FldName> <Size>11</Size> <ColName>WHOLELIFEINTERESTRATE</ColName> <FldName>WholeLifeInterestRate</FldName> <Size>4</Size> <ColName>WHOLELIFEINTERESTAMT</ColName> <FldName>WholeLifeInterestAmt</FldName> <InlineOccurs>12</InlineOccurs> <Size>8</Size> </Table> </VSAMVarFormat> <VSAMVarFormat> <TypeValue>T</TypeValue> <Table> <TableName>TERMLIFEPOLICY</TableName> <MaxRecSz>1320</MaxRecSz> <DataCols> <ColName>TERMLIFEAMOUNT</ColName> <FldName>TermLifeAmount</FldName> <Size>11</Size> <ColType>Int32</ColType> <ColName>TERMLIFETERMYRS</ColName> <FldName>TermLifeTermYrs</FldName> <Size>2</Size> </Table> </VSAMVarFormat> <VSAMVarFormat> <TypeValue>A</TypeValue> <Table> <TableName>AUTOPOLICY</TableName> <MaxRecSz>1320</MaxRecSz> <DataCols> <ColName>AUTOLIABILITY</ColName> <FldName>AutoLiability</FldName>

11 <Size>8</Size> <ColName>AUTOCOMP</ColName> <FldName>AutoComp</FldName> <Size>8</Size> </Table> </VSAMVarFormat> <VSAMVarFormat> <TypeValue>H</TypeValue> <Table> <TableName>HOMEPOLICY</TableName> <MaxRecSz>1320</MaxRecSz> <DataCols> <ColName>DWELLINGLOSS</ColName> <FldName>DwellingLoss</FldName> <Size>11</Size> <ColName>CONTENTSLOSS</ColName> <FldName>ContentsLoss</FldName> <Size>11</Size> </Table> </VSAMVarFormat> </VarFormats> </VSAMFile> </Files> </VSAMFiles> Generated T-SQL DDL IF OBJECT_ID (N'POLICYRECORD_TABLE', N'U') IS NOT NULL DROP TABLE [POLICYRECORD_TABLE]; CREATE TABLE [POLICYRECORD_TABLE] ( [POLICYNUMBER] CHAR(8) NOT NULL, CONSTRAINT PK_POLICYRECORD_TABLE_POLICYNUMBER PRIMARY KEY (POLICYNUMBER), [POLICYISSUEDT] INT, [POLICYTYPE] NVARCHAR, LastUpdate DATETIME NOT NULL, LastUser CHAR(15) NOT NULL ) IF OBJECT_ID (N'POLICYHOLDERDATA', N'U') IS NOT NULL DROP TABLE [POLICYHOLDERDATA]; CREATE TABLE [POLICYHOLDERDATA] (

12 ) [POLICYNUMBER] CHAR(8) NOT NULL, [Indx] INT NOT NULL, CONSTRAINT PK_POLICYHOLDERDATA_POLICYNUMBER PRIMARY KEY (POLICYNUMBER, Indx), [POLICYHOLDERBILLING] DECIMAL(8,2), [POLICYHOLDERPAYMENT] DECIMAL(8,2), IF OBJECT_ID (N'WHOLELIFEPOLICY', N'U') IS NOT NULL DROP TABLE [WHOLELIFEPOLICY]; CREATE TABLE [WHOLELIFEPOLICY] ( [WHOLELIFEAMOUNT] DECIMAL(11,2), [WHOLELIFEINTERESTRATE] DECIMAL(4,2), [WHOLELIFEINTERESTAMT_1] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_2] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_3] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_4] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_5] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_6] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_7] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_8] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_9] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_10] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_11] DECIMAL(8,2), [WHOLELIFEINTERESTAMT_12] DECIMAL(8,2), ) IF OBJECT_ID (N'TERMLIFEPOLICY', N'U') IS NOT NULL DROP TABLE [TERMLIFEPOLICY]; CREATE TABLE [TERMLIFEPOLICY] ( [TERMLIFEAMOUNT] DECIMAL(11,2), [TERMLIFETERMYRS] INT, ) IF OBJECT_ID (N'AUTOPOLICY', N'U') IS NOT NULL DROP TABLE [AUTOPOLICY]; CREATE TABLE [AUTOPOLICY] ( [AUTOLIABILITY] DECIMAL(8,2), [AUTOCOMP] DECIMAL(8,2), ) IF OBJECT_ID (N'HOMEPOLICY', N'U') IS NOT NULL DROP TABLE [HOMEPOLICY]; CREATE TABLE [HOMEPOLICY] ( [DWELLINGLOSS] DECIMAL(11,2), [CONTENTSLOSS] DECIMAL(11,2), )