%DWFK$&&(66WR $'$%$6%$$ E\ 6WXDUW%LUFK IURP,QIRUPDWLRQ'HOLYHU\ 6\VWHPV6RXWK$IULFD

Transcription

1 %DWFK$&&(66WR $'$%$6%$$ E\ 6WXDUW%LUFK IURP,QIRUPDWLRQ'HOLYHU\ 6\VWHPV6RXWK$IULFD 1

2 ,QWURGXFWLRQ O Objectives and Benefits O Applicable Environment O Terms and Definitions O System Components Objectives and Benefits Applicable Environment An ADABAS site setting up a warehouse using THE SAS SYSTEM. Terms and Definitions Some ADABAS terms that will be used throughout this presentation. System Components A conceptual overview of BAA 2

3 2EMHFWLYHV O Updating SAS without ADABAS interrogation O Automatic file translation O Efficient Initial Database Population O Continuous incremental updating O Automatic monitoring of file changes Initial Population: When creating a warehouse for the first time, it is difficult to identify all the correct columns from the start. This means many changes to your initial warehouse, which, if run on-line, would be devastating to your ADABAS environment. BAA gives you the capability to make these changes outside the ADABAS environment by using ADABAS dumps created by an ADABAS utility. Incremental updating: Update from ADABAS to the SAS Data Warehouse on an incremental basis thus avoiding reading the entire ADABAS file to obtain recent changes. Warehouse updating without ADABAS: Allow batch processing of ADABAS data using SAS without ADABAS executing. This avoids further processing on an already strained environment. Automatic File Translation: Automatically translate File Description Table(FDT) information to SAS variables. Automatic Monitoring of file Changes: ADABAS field changes can easily be identified and updated in the SAS register by using a utility program. FDT information can be read on-line by SAS/ACCESS to ADABAS or by using the ADABAS ADAWAN utility to unload these definitions. 3

4 %HQHILWV O Performance O Operational considerations O Maintenance O Incremental updating O Data normalization Performance: BAA reads ADABAS generated protection log (PLOG) tapes. These tapes are read without invoking ADABAS, obtaining at least an order of magnitude speed improvement over both other options. Operational considerations: At most sites running ADABAS, the databases are being utilized quite heavily, especially the overnight batch window. In contrast, the machine itself has spare capacity i,e, the bottleneck is ADABAS not the machine. It is this spare SYSTEM capacity that is used by BAA as ADABAS is not read at all. Thus causing minimal impact on the TOTAL performance of the machine, and to ADABAS, which must compete for resources. Maintenance: BAA allows for batch updating of translation formats and a registry ensuring that like fields are processed identically. NO PROGRAM CHANGES WHEN KEEPING OR DISCARDING AN ADABAS FIELD, just update the register Trickle feed updates: In cases where no reliable date-time stamps are kept (most often the case in ADABAS), reading the PLOG is the most efficient way of incrementally updating the warehouse. Data normalization: de-normalized database design is encouraged with The MU and PE field types. BAA allows the direct normalization of data by creating a relational table keyed by the UNIQUE file key 4

5 (QYLURQPHQW ADABAS UNIX PLOG SYSTEM The BAA was developed at a major short term insurance company in South Africa. The project was to use legacy ADABAS data bases to populate a SAS repository on a UNIX box. The products needed on the Mainframe are: SAS/BASE, SAS/ Access to ADABAS, SAS/Connect. The products needed on the UNIX are: SAS/ BASE, SAS/ Connect. The products needed on the PC are: SAS/ BASE, SAS/ Connect, SAS/ FSVIEW - to maintain the register. As the system makes use of Remote Library Services (RLS) and an access method supporting RLS is therefore necessary. 5

6 7HUPV FDT MU PE PLOG Register File description table Multiple value field Periodic group Protection Log BAA s register FDT ADABAS File Description Table. In SAS terms, the FDT holds the same information that can be seen by running a PROC CONTENTS against a data file. MU A de-normalization of ADABAS data where a field is allowed to repeat. PE A further denormalisation of ADABAS data where a group of fields are allowed to repeat. The fields in this group can themselves be MU s PLOG Protection Log generated by ADABAS. Register. BAA uses a register to merge your SAS and ADABAS attributes into a SAS dataset. This dataset, or register, is used as input to generate a SAS program to read the PLOG tape. 6

7 3URWHFWLRQ/RJV ADABAS Dataset 1 Dataset 2 Tape The ADABAS Operations Manual (section 5 for version 3.3.3) will detail the usage of Protection Logging. This section serves as an introduction the DUAL PROTECTION logging, which is the most common practice. Protection logging, PLOG for short, is used to ensure database integrity in the event of an ABEND of ADABAS. It is a record of all transactions against the database that causes an Add, Change or a Delete to any table on the DATABASE. A separate PLOG system is used for each active ADABAS nucleus or database. PLOGS are used as a record of activity against the database from a certain checkpoint that is known to be secure. For instance, ADABAS is backed up on Friday night just before the weekend batch window. It is close to month end and there are a lot of batch updates that go through. This is also a heavy period of on-line usage. On Thursday the following week a head crash on the disk renders your entire database unusable. Using the PLOG recovery system, it is possible to get back to the last PLOG dump that occurred just before the head crash. To achieve this, the secure backup of Friday night is restored to disk. ADABAS is then stared and used to read ALL the PLOG tapes that were generated from the backup to the crash. They are read in sequential order and in effect, all transactions are entered against the database in the order in which they happened just as before. The PLOG system will run anyway. Therefore using it to trickle-feed you warehouse will not incur any extra ADABAS overhead 7

8 6\VWHP&RPSRQHQWV O Register population O PLOG control file O Code files O System Process Populating the Register: This section will identify the automatic update capabilities from two sources: ADABAS FDT information and, if available, the SAS Warehouse information. Together, 95% of the register can be built without manual entry. PLOG Control file: An ADABAS file keeping information about the automatically generated PLOGs and an additional subset if necessary. A subset is used to keep only the PLOG records for the files that will be updated incrementally. CODE files: Using SAS/ACCESS, relatively static code tables can be downloaded when necessary, and are not processed using the PLOG method. System Process: This section describes a conceptual overview of the BAA system. 8

9 5HJLVWHU3RSXODWLRQ O Conceptual Overview O Manual Options:» Redefined fields» MU s and PE s» Code conversions» BAA system components Introductory Slide 9

10 5HJLVWHU3RSXODWLRQ ADABAS ACCESS ADAWAN SAS register Flat files SAS program Register Population: Up to 95% of the register can be sourced from ADABAS and the SAS Data Warehouse (if it exists already). An ADABAS utility called ADAWAN can be run against ADABAS to output, in flat file format, the FDT (File Description Table) for all of the relevant ADABAS tables. The FDT information is equivalent to the SAS PROC CONTENTS information. 10

11 5HJLVWHU3RSXODWLRQ SAS register PROC contents DATA WARE- HOUSE SAS Attributes: The Register is maintained on-line by means of a simple FSEDIT screen. SAS Attributes such as Variable Name and Table Name are added at this point. If the SAS warehouse already exists and the BAA system is being installed post warehouse population, the SAS attributes can be merged onto the Register to avoid re-typing the information 11

12 5HJLVWHU3RSXODWLRQ 0DQXDO2SWLRQV O Redefining fields Redefining fields: Sometimes the ADABAS field is a concatenation of sub-fields and is redefined in the Natural program manipulating this data. This same capability exists at the DATA level for population purposes, making your UPDATE programs less complex. 12

13 5HGHILQLQJ)LHOGV Claim ID: Claim No. Section Item Example: Claim ID on ADABAS is a concatenation of 3 separate fields In SAS these fields will be stored as Claim Number, Section and Item. This functionality would be done in a Natural program before manipulating the data. Here, the redefinition is done at the data level. 13

14 5HJLVWHU3RSXODWLRQ 0DQXDO2SWLRQV O Redefining fields O MU s and PE s MU s and PE s: Two ways to handle MU's and PE s exist. If the NATURAL program updating an MU has a fixed number of occurrences and each occurrence is always be filled in, then the MU can be immediately expanded by the user entered number of repeats. 14

15 08ªV Denormalised Data (MU) Name Salary Address Smith Bn Row Redmond Washington ADABAS ACCESS Name Salary Addr-1 Addr-2 Addr-3 Smith Bn Row Redmond Washington SAS e.g.. Address is an MU that always repeats 3 times. Address is registered as ADDR in the register with the value 3 in the Occurs field ( a count of how many times a field will repeat). Addr_1, Addr_2, Addr_3 and Addr_4 will be the resulting columns on the SAS dataset. The other option is to create a separate relational table with the value of Addr repeating over four observations. This table will be keyed by the user specified KEY on the register. In this case, the number of repetitions will be read from the PLOG tape at run time. 15

16 3(ªV Denormalised Data (PE) Name Salary Children DOB Smith Mark 1/2/72 Andrew 19/8/74 ADABAS ACCESS Name Salary Children DOB Smith Mark 1/2/72 Smith Andrew 19/8/74 SAS Example: An example of a PE. There is only one way to handle a PE, which is the same as the second option for MU s. 16

17 5HJLVWHU3RSXODWLRQ 0DQXDO2SWLRQV O Redefining fields O MU s and PE s O Code conversions 17

18 &RGH&RQYHUVLRQV O Classification variables kept as CHAR O Analysis variables kept as NUMERIC O Conversion at DATA level O Including or excluding leading zeros Classification Variables: For usage and display purposes, classification codes are kept as character. This avoids confusion when using utilities such as SAS/ASSIST by end users. Conversion at data level and leading Zeros: As with redefined fields, Numeric to Character conversions happen at the data level. This involves setting an indicator on the register, including setting up leading zeros. 18

19 6\VWHP&RPSRQHQWV O Populating the register O PLOG control file 19

20 3/2*&RQWURO)LOH PLOG ADABAS CONTROL FILE SAS/ACCESS or batch job subset PLOG The PLOG Control file: The PLOG Control file is the operational interface between the ADABAS protection Log System and your SAS data warehouse. Two PLOG datasets exist on disk. When the first dataset is filled up, ADABAS will begin logging to the second dataset and call for a TAPE to dump the information in the first dataset. At this point a second job may be set up to filter out only the ADABAS file in which we have an interest and write this information to a second tape. This second tape must be part of a GDG tape system to avoid accidental erasure. The PLOG control file is now updated with the relevant information and a STATUS of NEW. When this control file is accessed from the SAS system the STATUS is changed to UPDATE. An unsuccessful completion will result in an ERROR STATUS. 20

21 6\VWHP&RPSRQHQWV O Populating the register O PLOG control file O Code files 21

22 &RGH7DEOHV O Static Code files replaced O Bypass the PLOG Update O SAS/ACCESS or Batch Code Tables: Static ADABAS files are copied across by means of SAS/ACCESS or by dumping them to flat files and reading them into SAS via a batch job. 22

23 6\VWHP&RPSRQHQWV O Populating the register O PLOG control file O Code files O System Process System Process: This section describes a conceptual overview of the BAA system. 23

24 6\VWHP3URFHVV Generated Program PLOG Register Control File Driving Macro UPDATE FILES & TABLES WARE- HOUSE Driving Macro: At the core of the BAA system is a Driving Macro. It is kicked off by a batch job. Its will : read the control file for unprocessed PLOGS read the register and create code which will call for and read the PLOG tape create update datasets including derived columns modify (in-place) the SAS data warehouse The SAS modify in place is used to apply any additions/changes/deletions to the base data warehouse. This requires the use and maintenance of unique indices on the warehouse. UPDATE IN PLACE will provide a speed improvement when applying relatively small dataset to a relatively large dataset keyed on a unique index. 24

25 %HQHILWVRI%$$ O Time spent at setup not in maintenance O No month end crunch O Data available daily vs. monthly O Much less mainframe resource used O SPACE BAA takes time to set up. This time is spent upfront with a large time-saving at each and every month-end (or whatever population cycle you have). This means that there is no month end crunch to get the data. All attention can be paid to delivering reports. Since the warehouse is updated regularly, the data is current BAA does not cause any extra ADABAS CPU seconds to be used. This means savings in actual resource used as well a less strain placed on the over-night batch window which is generally quite full. There are no programs to maintain. Maintenance is reduced to keeping your Register up to date. Modify in place reduces the SPACE requirement on the SAS side. There is no extra disk space required on the mainframe since the data resides on PLOG tape which would be used anyway. BAA handles denormalised ADABAS data. BAA handles the redefinition of fields at the data level BAA handles code conversions at the data level BAA handles the insertion of decimal points at the data level 25

26 %HQHILWVRI%$$FRQW O denormalised data O redefined fields O code conversion O insertion of decimal points 26