Moving a Large SAS System Into an MVS Production Environment The Trials and Tribulations

Transcription

1 Moving a Large SAS System Into an MVS Production Environment The Trials and Tribulations Bernard Gardocki and Stephen Rhoades IMS HEALTH Inc., Plymouth Meeting, Pa ABSTRACT It is not the role, or responsibility of production integration personnel to review and test each program transitioning to the production environment. Therefore, the developer s implementation of systematically computed control totals becomes an essential requirement to successfully implement programs into the production environment. So how do you do move a system into production? We will cover the process of passing parameters into SAS streams, verifying record counts from tape files and disk files. Providing reports of record counts and reports on why a job abended and what should be done. A number of different techniques are used to check on input files including dictionary files and CALL EXECUTE, the code and the reasons they were used will also be covered. Definition of a large system. A database of 1,600,000 cases. Over one hundred job streams, many run multiple times per cycle. Files with up to 260 variables. Over 500 different output files. 8.3 million new records added quarterly. The system now has data for ten years. The quarterly update process takes weeks to process. The process ties up large portions of the CPU. Quite a few jobs run around an hour or more of CPU time PROBLEMS, PROBLEMS, PROBLEMS AND SOLUTIONS! For the SAS code, any comments in this color were inserted to add clarity. Problem 1. Reading in foreign tapes. Ever receive a box of poorly labeled cartridges containing multiple flat files each of which spans multiple cartridges? We have! How did we handle it? Our amateur cryptologist can usually mix and match the tapes. However, this is not very effective when we receive last quarter s data again for this quarter, or we are missing tapes or have received the wrong tape despite what the label says. We need to know as quickly as possible that there is a problem with the new data. Solution to Problem 1 No, we did not get a new supplier! However, we now receive written record counts, which are used for control record totals. The number of records on the cartridges must match the control totals or the process produces an error report and stops until the record count problems are resolved. This is how we set the job up to check the incoming record counts. PARTIAL JCL: //* //*- Read new external data - //*- Below SYSPARM allows passing parameters - //* //S03A EXEC SAS8, // PARM='SYSPARM="&YR1&YR5&QTR"' Multiple external files are read into the system using GDG file types //I1PHS1 DD DSN=NESUG.LQ00051.ABC,DISP=SHR // DD DSN=NESUG.LQ00051.DE,DISP=SHR, // UNIT=AFF=I1PHS1 //* Quarterly Rec counts for incoming files

2 Each quarter NESUG001 will be edited and control total record counts will be entered. //I2CNTS DD DSN=NESUG.PARMLIB(NESUG001), // DISP=SHR //O8ABEND DD SYSOUT=* If rec counts do not match an error rpt is sent here THE SAS CODE: Macro ERRABND2 is used later in the program. It will abend the job if the record counts on the external file being read do not match the control total record counts. %Macro ERRABND2; /*Start Macro*/ INRECS is calculated by the job. RECCNT is the rec control total contained in NESUG001 and passed into SAS using SYSPARM. If the record counts are not equal write the error report to O8ABEND %If %EVAL(&INRECS) ne %EVAL(&RECCNT) %THEN %DO; NOTE: The error report must be written prior to abending the job. DATA _NULL_; File O8ABEND; Astr=repeat('*',62); '0667' ; ' '0667' ; Additional report code removed to save space in this poster. Abend job with a return code of 667 ABORT ABEND 667; %End; %Mend errabnd2; /* End Macro */ Run; DATA O1CLM.abcfil O2MH.defil ; infile I1PHS1 END=EOF; If an external file is empty, must place EOF and N=1 prior to the INPUT statement. /* */ /*- Are input files empty? -*/ /* */ IF EOF and _N_ = 1 then do; CALL EXECUTE('%ERRABND3'); END; groupnum ssn_num key_memb Increment INRECS for each obs read into program. INRECS+1; /* Increment rec count */ When the last record is read into SAS use SYMPUT to move record count variable, INRECS to a SAS Macro variable of the same name. If EOF THEN DO; CALL SYMPUT('INRECS',INRECS); CALL EXECUTE runs macro ERRABND2 to compare the actual number of recs read with the control number of recs, abending if there is not an exact match. NOTE: Another Data Step runs while still it this step. CALL EXECUTE('%ERRABND2'); End; To insure the data is correct the source type field is checked. Processing stops if not correct. Here s how this is done: If typeflag in ( A, B, C, D ) Then someday 125 my_prince 135 will_come $char10.; else do; Send a report and abend the job. ERRABND1 code not shown

3 srctype=typeflag; CALL EXECUTE('%ERRABND1(' srctype ')'); end; Problem 2. A paradox, this year s data is not on last year s tapes Every quarter a number of jobs read in the output of the same jobs for the previous quarter. There is a problem for the first quarter processing which is expecting a SAS data file for the current year on last year's file. This file does not exist, how do we handle this? Solution to Problem 2 We opted to handle this through a combination of SAS and JCL. We used SAS to pass a return code of 1 to the system if we were processing the first quarter of the year for the most current data year. To do this a SAS program was written to pass a return code back to the JCL to alter processing. There is no manual intervention; the system does this automatically. To do this we can use SET statements in the JCL to create what look like SAS macro variables. These allow us to change things like dates and file names then pass them directly into the SAS programs. Here s how it goes: SET statements are used to dynamically change values within a JCL stream. Follow QTR and FLNBR as they get moved into the SAS code // SET QTR=1 CURRENT QTR // SET AFILE='ABC' SAS file name // SET PQTR=4 Previous QUARTER // SET PYR='1999' Year for prev quarter // SET DYR='2000' Data yr // SET FLNBR='1' File nbr, this is yr 2000 //*Set a condition code, if the quarter is 1, //* the return code is set to 1. &QTR and &FLNBR, from the SET statements, are passed into the SAS program. //S03A EXEC SAS8,PARM='SYSPARM="&QTR&FLNBR"' //********************************************* ***** //SYSIN DD DSN=G.SRLIB, DISP=SHR, // SUBSYS=(PANV,,NESUGU191) The SAS Code looks like this: DATA _NULL_; LENGTH QTR $1 FLNBR 1; /*Read in the SYSPARM values*/ QTR= SUBSTR(SYSPARM(),1,1); FLNBR= SUBSTR(SYSPARM(),2,1); IF QTR= 1 and FLNBR=1; abort return 1;/* Set rc to 1, end program*/ This JCL runs when the Return Code is 0 // IF RC LT 1 THEN //* Processing qtrs 2, 3 or 4 as the RC is 0 //* Copy SAS file temporary disk space //S05A EXEC SAS8,PARM='SYSPARM="&AFILE"' Read prev qtr file, write temp file //I1IN DD DSN=M&PYR&PQTR..A&DYR, // DISP=SHR //O1OUT DD DSN=&&TEMP01, // DISP=(NEW,PASS,DELETE), // SPACE=(CYL,(3100,25)),UNIT=DA //SYSIN DD DSN=I.SLIB,DISP=SHR, // SUBSYS=(PANV,,NESUGU192) Notice the ELSE. This JCL runs when the Condition Code is 1 // ELSE //***************************************** //* Processing the 1st quarter *** //* Create a dummy file *** //***************************************** //S01A EXEC PGM=IEFBR14 //* //O1A DD DSN=&&TEMP01, // DISP=(NEW,PASS,DELETE), // SPACE=(27648,(1,1),RLSE),UNIT=DA

4 ENDIF tells the system to run the rest of the JCL sequentially // ENDIF Problem 3. Adjusting memory A number of these jobs use large amounts of memory. We often boosted the jobs to the maximum allowed on our system, that is 128Mg. These will be tuned over time. As the SAS code is protected a way to dynamically change the memory option MEMSIZE was required. Solution to Problem 3. To solve this a JCL SET statement is used to change the memory and region sizes on the fly and pass it into the SAS program. First we use a JCL SET statement: // SET REG='64M' Notice that &REG goes to both REGION and MEMSIZE. //S15A EXEC SAS8,REGION=&REG, // PARM='MEMSIZE=&REG SYSPARM="&DYR&YR&QTR&FLNBR&PY R"' Problem 4. Spanning disk packs Spanning disk packs in an MVS shop running SMS is a rather onerous task. With SAS 6 we had to get create permanent temporary work files on private disk packs and then remove them after the jobs completed. This meant that we were severely limited on the number of jobs we could run simultaneously and other jobs needing the same space were subject to failure. Here s the code that we used with SAS6. All use this DCB and Space info: DCB=(DSORG=PS,RECFM=FS,LRECL=27 648, BLKSIZE=27648), SPACE=(CYL,(3000),RLSE), UNIT=SYSDA, //S01A EXEC PGM=IEFBR14 //********************************************* //* Create temp work space for SAS ** //********************************************* //DD3 DD DSN=NE190,DISP=(,KEEP), // VOL=SER=NESUG010 //DD5 DD DSN=NE190,DISP=(,KEEP), // VOL=SER=NESUG009 //DD6 DD DSN=NE190, // DISP=(OLD,CATLG),UNIT=SYSDA, // VOL=SER=(NESUG010,NESUG009) The NE190 file is then used in the SAS program as shown here: //S15A EXEC SASP,REGION=128M, // PARM= // 'SYSPARM="&DYR&YR&QTR&FLNBR"' //WORK DD DSN=NE190,DISP=(OLD,DELETE) Solution to Problem 4 (NOTE: SAS 8 is needed for this) No IEFBR14 utility is needed for this SAS 8 solution. //S15A EXEC SAS8,REGION=&REG, // PARM='MEMSIZE=&REG SYSPARM="&DYR&YR&QTR&FLNBR&PY R"' The WORK file below spans public disk packs without the pain. //WORK DD UNIT=(SYSDA,3),SPACE=(CYL,(3200,3200 ),RLSE) Problem 5. Reading SAS files from tape. The code we will be examining is used to check that tape SAS library files contain SAS data files with observations. The number of observations is not stored on tape SAS files. For our system this means that we can find out if a SAS data file exists, but we need to read the entire file to determine the number of records it contains. We need to know both.

5 Solution to Problem 5 We need to check that the SAS data file exists and the number of records matches our control numbers. Here is how it is done. Macro EXIST checks that the file exists and CHKSEL provides the rec counts once the file is read. %MACRO EXIST(DSN); %GLOBAL EXIST; %IF &DSN NE %THEN %STR( DATA _NULL_; IF 0 THEN SET &DSN; STOP; RUN; ); %IF &SYSERR=0 %THEN %LET EXIST=YES; %ELSE %LET EXIST=NO; %MEND EXIST; Alternative code for checking the existence of a SAS data file. %Macro IsItRl(DSN); %IF %SYSFUNC(EXIST(&DSN)) %THEN %DO; %PUT &DSN Exists; %END; %ELSE %DO; %put &DSN does not exist; %End; %Mend IsItRl; Run; This program reads in the entire SAS file from tape to disk. The CHKSEL macro provides the record counts. For those of you unfamiliar with SAS dictionary files, they store SAS file information within them. Macro CHKSEL takes advantage of these features. NOTE: The names being passed into the macro must be capitalized. MACRO CHKSEL(LIBNAM,MEMNAM); Proc SQL; Select nobs /*file obs counts*/ Into :reccnt /*macro var*/ From dictionary.tables where libname="&libnam" and memname="&memnam"; Quit; The next section compares the number of observations (&RECCNT) against a control total (&I4RECNT) Data _NULL_; IF &RECCNT = &I4RECnt Then do; End; Else Do; CALL execute('%errabnd1(' "&RECCNT" ')'); End; %MEND CHKSEL ; Read in the new quarterly from tape and test to see if the file exists by using Macro EXIST. %Macro Datanew; %EXIST(I1PHSI.&NAFILE); /* file exist? */ %IF &EXIST =NO %THEN %DO; NO means the SAS data file is not there, print an error report and terminate the job. DATA _NULL_ ; length badfile $ 23; bx="i1phsi.&nafile missing"; /*SAS file does not exist, abend job*/ CALL EXECUTE('%ERRABND4(' bx ')'); %END; %ELSE %DO; /*The file exists*/ /* */ /* SELPATID determines which patid recs -*/ /*- to process from the previous qtrs -*/ /*- New is all data sent to us for the qtr. -*/ /* */ DATA NEW SELPATID (KEEP=PATID); Here is a problem that was encountered using SAS version 8. Notice the drop=origin on the SET statement. It raised havoc with variable labels. SAS Institute issued a zap to correct the problem of 5/18/2000. SET I1PHSI.&NAFILE (rename=(patid=patido ssn_sub=ssn_subo) drop= origin) END=EOF; %END;

6 /* */ /* With tape, the entire file needs to be -*/ /* read and then rec counts compared -*/ /* */ After the complete file is read compare against the control totals %CHKSEL(WORK,SELPATID); %Mend DataNew; Run; %DataNew; Problem 6. Reading SAS 8 files into a SAS 6 session. Early in our testing we did this by accident. If part of your salary is based on the amount of CPU your jobs use, with the more the better, then this is the way to go. We found that our jobs always used the maximum amount of CPU time before the infamous 322 return code appeared and the jobs stopped running. Solution to Problem 6 Don t do this!!!! Conclusion There is a sign over the Chicago Police Department s training facility, it states The more you sweat here, the less you bleed out there. Our motto is The more internal checks you have in your production jobs, the less chance you will be sweating in your boss s office. Writing SAS code to make sure the numbers that are generated are correct, can be tedious and nerve wracking, but in the end it will be worth it. The solutions and suggestions in this paper should make this process less tedious and less nerve wracking for you. REFERENCES: 1. SAS Guide to Macro Processing, Version 6, Second Edition. Pg. 265 CONTACT INFORMATION: Steve Rhoades & Bernie Gardocki IMS HEALTH Inc 660 W. Germantown Pike srhoades@us.imshealth.com bgardocki@us.imshealth.com