Managing complexity in large SAS system applications John Niss Hansen, HAFNIA ( Denmark)

Transcription

1 Managing complexity in large SAS system applications John Niss Hansen, HAFNIA ( Denmark) The paper will address problems in large SAS applications, where data from many sources are extracted periodically from a constantly changing and sometimes even turbulent production environment and placed in both SAS and DB2 tables. Objectives of application management are achieved through an active data dictionary based on the Relational data model and a macro execution facility based on the data flow principle. The system is managed from a version 6 SCL application for monitoring the history of execution and for updating the system definition and documentation. The application was born with the success of single tabulate, based on a SAS dataset easily created from a few tapes with some minor data fiddling. The success should soon be repeated just with new figures and little column more. Heard the story? Well after a while you have several sources of data, many SAS libraries, format libraries, source and JCL libraries, catalogs etc. So what? Here are some headlines for the next few chapters. 1. Increasing amount and diversity of data. 2. From adhoc to periodic update. 3. Handling breakdowns. 4. Handling changes and consistent definitions. 5. Documentation. 33

2 1. INCREASING AMOUNT AND DIVERSITY OF DATA. It's a problem to keep track off all the names, fysical and logical representations and how they are interrelated. Data flows in and out of the system and even BDP professionals have problems to fully comprehend the system. The none professional trust the professional, -which mayor especially may not have written something down. In order to get rid of this mess, we have made a data dictionary. It contains information of all SAS libraries, what members they contain, which variables are on each member. All variables are declared only once in a special sasdataset. In addition to this each member are declared with a primary key, that is the set of variables that determines each observation uniquely, which in fact turns each member into a relation in accordance with the relational data model. The definition of a new member is always entered in the dictionary before the actual creation. In fact the dictionary supports the member creation through a macro call. In addition to have operational support, we can now answer questions about where the data are and what they mean. A variable are defined the same way everywhere and can also be changed (fx the label or the length) from a single place. The data dictionary is maintained interactively through SAS/AF + FSP and SCL., J. 34

3 2. FROM ADHOC TO PERIODIC UPDATE. From the first adhoc jobs to the later periodic executions, you may observe that some additional bookkeeping has entered the stage. Files are no longer that distinct, they now comes in series with generation number or otherwise codified names and with running creation dates. As long as your application depend on only of a few of these, you keep an eye with the file catalog and watch the proper arrivals and then submits the jobs with only a few changes. A few generation datasets more and you are in trouble, especially if you want some hollidays also. There becomes too many things to keep track of. A solution here wold be to let the operations staff take over, with tools like OPC or UCC7. But here I have experienced problems in how to specify what it is you want. Every installation has its own manual that in less than 500 pages describe how you document your system in detail. The casual SAS user, can easily use a week on this inter price. The next problem is when he wants to change the programs. Here security and other procedures, may slow down the humble SAS programmer. What we have done is to make frequent lookups in the file catalog via the IDCAMS utility. This gives transactions for maintaining the history of changes in the file catalog as a SAS data set. Here we can see files turning up and others leaving the stage. Macros can then wait for the proper file(s) to turn up and then be released for execution. This is the Data flow principle. Here we have to keep track of which macroes reads which files, but instead of describing this via JCL, we use a SAS dataset for the relation and the filename/libname statement for the dynamic allocation. In this way we spare both JCL and job submission systems like OPC. On top of it, we get significant information about the application in tabular form, serving the purpose of dynamic documentation. 35

4 3 HANDLING BREAKDOWNS. One of the more timeconsuming task, is handling breakdowns. Here is a list of the most frequent causes of abends. Program errors Shortage of disk space Tapemount rejects Collision with other users during updates Program errors, due to lack of proper test facilities, lack of time or lack of good luck. Whatever the reason was, you correct the bad statements and rerun the job. If the job is a large one, it may have done a lot of good work before it went down and therefore you may consider an adhoc version of the job where the more succesfull part has been cut away. This job is a typical onetime job and a good candidate for yet another failure. What we have done is to have macro execution facility, that orders the execution of macros. Keep track of which macros were executed succesfull and performs a rollback for those that abened. There is only two jobs that performs the actual execution. The first is the master that do all the bookkeeping, the second will take the macros found ready for execution one at a time, allocate the necessary files, execute the macro and then report the success. If the second job went down, it is simply resubmitted. It will then perform the rollback, set the bad macro aside and find another macro that is ready and then go on with that one instead. To reach such a level of flexibility we have added several things. A macro is not just a bunch code, its an object that can be in se veral different states, even without being executed. To keep explicit track of these states is one of the secrets. Here are some of them. Waiting for time. A macro that runs one time each month, will be in this state for the rest of the month after the succesfull execution. Waiting for data. When we reach the first day in a new month, the time has come to see if the new data has come. Ready for execution. Here all the input data is in place. Active. Here the macro is actually executing. Succesfull end. The macro has just fulfilled its job and will soon be waiting for time again. 36

5 Abended. The macro missed and the whole thing went down. Someone have to take a look at it, correct it, and restart it in the macro state waiting for data. In the meantime many other macros may run, except those that use the data that the abended macro should have updated. The system will take care of that. Describing a macro as an object opens up for the introduction of other state variables as fx. priority and schedulation frequency. Back to breakdowns. Shortage of disk space is another frequent reason for abends. When this happened the procedure is almost the same as for programming errors, except that we may choose to optimize for space usage instead of correcting an error. By systematic collecting of vtoc information for our SAS libraries, we can in fact predict potential breakdowns and to some degree prevent this sort of abends. The last two reasons of breakdowns, shows up unpredictable as you want access to some data. If the data is on tape you may get a tapemount reject due to unawailable free tape units or you may want to update a SAS library that is already in use by another user. What we do here is that we wait nonbusy a few minutes and then try again. If we after some trials still can't get in touch with the data, we perform a rollback and exit SAS without abending. The macro will be schedulated again at a later time, without any human intervention. 37

6 4 HANDLING CHANGES AND CONSISTENT DEFINITIONS. When edp systems grow large, they become increasingly difficult to change in a safe and consistent manner. When you make changes in one place, something may go wrong later at some other place. Traditional documentation is often missing or unreliable due to its separation from the operational system it is supposed to describe. When you create new members in sas libraries, how do you assure that variables are spelled the same way, have same (minimal) length and type as in other places? Do you alays put on a label? If you decide to delete a variable from a SAS dataset, how do you find the programs that may be affected? Being able to reuse old variable definitions in new table definitions give us a rich opportunity for consistency in use of names. With the data dictionary and the macro execution facility we have most of what is needed to analyze the relationships between tables and macros (programs). The data is operational and therefore also up to date. The question we may ask can be answered with simple programs. Often used queries are embedded in SCL under SAS/AF. A query like 'Which macros could be affected if i change the macro named POLICER.' can be answered graphically (see graph on last page). The same graph may also apply to the query: 'Which macros should possibly rerun in case the macro POLICER should rerun.'. 38

7 , 5 DOCUMENTATION. Also the old 'infant terrible', written documentation, has got new possibilities with a system like this. In case of documentation for the user, we produce a table manual of the more than 200 SAS datasets and more than 40 DB2 tables. The document is more than 300 pages of generated SCRIPT. The manual comes every second month. When old users see the manual they immediate turn the index of changes, to see what has happened since that date when they last retrieved a documentation. Should anyone be excited by or have experience with a system like this, don't hesitate to contact me or one of my colleagues. John Niss Hansen Hafnia Insurance Holmens Kanal Copenhagen Denmark. t f t,. r f t f i t I f : J r. i \ 39

8 Dependencies of the macro POLICER KUNDISIC J [1 PORTMAN VlRTDDTA J ' t :...,. i VlRTDDEA USERKBONI VIRTDDPA I., " ;.J { POLPROD ERSTAFSL f... r-----i VIRTDDSA I : k i. f:: IpOUCER KWIP I REGNSI<AB VIRTDDRA I <-,.,' \' ':: 'J,',, 1- " I 5 f ii j, ' :' J; t " " 1: ". I I I I.; :. BESIGllG J TAPEPOL I ACTOMKOD I MANAGE VlRTDDOA I POUCESP PORTBEV VIRTDDQA I Each box is a macro. Each rlll8 is a stream of data., '1: j; Ii: 40