High Performance Relational Implementation of VSAM Indexed Files

Transcription

1 High Performance Relational Implementation of VSAM Indexed Files Note: this discussion applies to Microsoft SQL Server, Oracle Database and IBM DB2 LUW. The organization of a SQL table with its rows and columns is significantly different than the organization of a VSAM indexed dataset (also known as a Key Sequential Data Set or KSDS). A VSAM dataset has a primary index with a unique key. In addition, it can have one or more alternate indices which may or may not allow duplicate keys. Index Path Problem A single record can be randomly read via any index key or multiple records can be read using a start operation followed by multiple read next and/or read prior operations. A start on an index sets the index pointer position and allows read next and/or read prior operations from that point in index key order up to the end or beginning of the file. SQL cursors are not sufficient for implementing this functionality for a number of reasons. One reason is read priors can go before the point of the initial start. Another reason is that use of cursors can generate a very large cursor fetch prior to reading the first record -- a situation that can adversely affect performance. However, this continuous sequential reading via key order can be efficiently implemented using SQL Windowing with OVER-ORDER BY clauses and row numbers. Our tests on a file of one million records show instant start and read next/prior operations at any point in the index. Transaction Rollback Problem Since VSAM update operations are now implemented in a relational database, VSAM updates can be part of a SQL transaction which means that multiple updates can be either committed or rolled back as a group. VSAM updates use an update connection which can have an associated transaction. When the update connection has an associated transaction, that connection is kept open and used for all updates until it is closed via a commit or rollback. Read (non-update) operations are performed using a read-only connection which is closed after each operation. Record Field Problem Also, a VSAM file consists of records described by one or more COBOL 01 record descriptions. While VSAM records are similar to SQL table rows, the record s COBOL fields are not similar to SQL columns because a COBOL record usually contains groups, redefines and occurs which are not possible in SQL columns. To avoid these issues the

2 entire record is stored in one binary column of size equal to the size of the record. The maximum size of a SQL Server binary column is 8000 bytes. Records larger than this can be accommodated using either multiple record data columns or binary large objects. In addition to storing the data in a binary column, the record s key values are duplicated as key columns and used for indexing the table. High Performance Implementation This implementation of VSAM datasets in a relational database is designed to achieve the fastest access times possible. For a discussion of access time, please see our Why is SQL emulation of VSAM slow? document. The tradeoff for maximum performance is a limited ability to normalize the SQL database and to access it as a pure SQL database from newly written applications. For a discussion of LPS advanced normalization capabilities, please see our Normalized Relational Database Implementation of VSAM Indexed Files document. For this implementation of VSAM files using SQL Server tables and columns, each VSAM file is first mapped to a SQL table containing the following columns: 1. Primary key. 2. Alternate keys (if any). 3. Record data in a byte array (RecData). 4. Date and time last updated (LastUpdate). 5. User who last updated this record (LastUser). These table definitions are automatically generated from an XML description of the VSAM files. The XML file definition is generated from parsing IDCAMS definitions and CICS File Control Table entries. Below is the generated T-SQL to create the tables for the Video Rental demo VSAM files. CREATE TABLE VIDCUST ( CUSTNUM CHAR(7) NOT NULL, CONSTRAINT PK_VIDCUST_CUSTNUM PRIMARY KEY CLUSTERED (CUSTNUM), RecData BINARY(123) NOT NULL, LastUpdate DATETIME NOT NULL, LastUser CHAR(15) NOT NULL) CREATE TABLE VIDMAST ( VIDNUM CHAR(8) NOT NULL, CONSTRAINT PK_VIDMAST_VIDNUM PRIMARY KEY CLUSTERED (VIDNUM), CUSTNUM CHAR(7) NOT NULL,

3 RecData BINARY(57) NOT NULL, LastUpdate DATETIME NOT NULL, LastUser CHAR(15) NOT NULL) CREATE NONCLUSTERED INDEX IX_VIDMAST_CUSTNUM ON VIDMAST(CUSTNUM) Once the SQL tables have been create in the database, the following methods are available to perform all the functions of a VSAM file including randomly reading a record, writing a new record, updating or deleting an existing record, start a browse and read next or read prior records. Each VSAM-SQL method returns one of the following results enumerations depending on the type of operation: a. Ok Operation completed okay. b. NoRecord No record found. c. ReadNormalKey Record found. d. ReadDuplKey Record found and duplicate key record follows (CICS only) e. ReadBOF Read prior past beginning of file f. ReadEOF Read next past end of file. g. ErrDuplKey Write or update caused duplicate keys on unique index. h. ErrNoStart Attempted read next or read prior with no previous start. i. ErrLastUpdate Update record was changed from prior read. j. ErrUnknown Unexpected exception encountered The set of VSAM-SQL methods and their parameters are described below. Using these methods we have implemented all the VSAM EXEC CICS commands and all the COBOL VSAM file verbs. As part of testing, we ran the VSAM Video Rental demo using this VSAM-SQL interface. 1. Read Record via Key ReadKey(VSAMKeyInfo KeyInfo, enumkeycompare Relop, string KeyVal, int KeyLength, bool CICS, out byte[] Rec) a. KeyInfo specifies key offset, length and type b. Relop specified key relational operator (EQ, GE, GT). c. KeyVal specifies the key value for the read d. KeyLength specifies a partial key read e. CICS, when true, specifies the read is for CICS which may require a read ahead to determine if the next record contains a duplicate key f. Rec is the returned record data

4 2. Write a New Record : Write(VSAMFile File, byte[] Rec) a. File specifies file and key info b. Rec record data to write 3. Rewrite an Existing Record : ReWrite(VSAMFile File, byte[] Rec) a. File specifies file and key info b. Rec record data to rewrite using primary key 4. Delete an Existing Record Delete(VSAMKeyInfo KeyInfo, string KeyVal, int KeyLength, out int NumRecs) a. KeyInfo specifies delete key using equal relational operator b. KeyVal specifies key value c. KeyLength specifies delete using partial key d. NumRecs returns number of records deleted 5. Start Browse Operation Start(VSAMKeyInfo KeyInfo, enumkeycompare Relop, string KeyVal, int KeyLength, out VSAMBrowse Browse) a. KeyInfo specifies index to be used for the browse operation b. Relop specified key relational operator (EQ, GE, GT) c. KeyVal specifies the key value for positioning index pointer d. KeyLength specifies browse using partial key e. Browse return a browse object for use in read next and read prior operations 6. Read Next or Prior Record in Browse ReadNext(VSAMBrowse Browse, bool CICS, out byte[] Rec) ReadPrior(VSAMBrowse Browse, bool CICS, out byte[] Rec) a. Browse is the browse object obtained from a previous start browse b. CICS, when true, specifies the read is for CICS which may require a read ahead to determine if the next record contains a duplicate key c. Rec is the returned record data 7. End Browse Operation : EndStart(VSAMBrowse Browse) a. Browse is the browse object obtained from previous start browse

5 8. Remove All Record : Remove(VSAMFile File) a. File specifies file and key info (used when opening file output) VSAM Files Information When using VSAM-SQL, programs need to know certain information about the VSAM files. That is the purpose of a set of classes that contain this data. This information is gathered by parsing the IDCAMS VSAM file definitions and also parsing the CICS File Control Table. The VSAM info classes are: 1. VSAMFiles is a list of all VSAM files and several associated dictionaries for rapid searching by CICS FileId or DSNName. 2. VSAMFile contains info for a specific file including its record size and a list of its keys. 3. VSAMKeyInfo contains info on a file index including its CICS FileId, DSNName, record offset and length and if duplicates are allowed. Below is the serialized XML for the two Video Rental demo VSAM files. <?xml version="1.0"?> <VSAMFiles xmlns:xsi=" xmlns:xsd=" <Files> <VSAMFile> <FileOrg>vINDEXED</FileOrg> <MaxRecSz>123</MaxRecSz> <VSAMKeyInfos> <VSAMKeyInfo> <FileId>VIDCUST</FileId> <DSNName>MSS.PKS.CODE1.VIDCUST</DSNName> <ColName>CUSTNUM</ColName> <Offset>0</Offset> <Length>7</Length> <Dupls>false</Dupls> </VSAMKeyInfo> </VSAMKeyInfos> </VSAMFile>

6 <VSAMFile> <FileOrg>vINDEXED</FileOrg> <MaxRecSz>57</MaxRecSz> <VSAMKeyInfos> <VSAMKeyInfo> <FileId>VIDMAST</FileId> <DSNName>MSS.PKS.CODE1.VIDMAST</DSNName> <ColName>VIDNUM</ColName> <Offset>0</Offset> <Length>8</Length> <Dupls>false</Dupls> </VSAMKeyInfo> <VSAMKeyInfo> <FileId>VIDMSTA</FileId> <DSNName>MSS.PKS.CODE1.VIDMSTA</DSNName> <ColName>CUSTNUM</ColName> <Offset>40</Offset> <Length>7</Length> <Dupls>true</Dupls> </VSAMKeyInfo> </VSAMKeyInfos> </VSAMFile> </Files> </VSAMFiles>