CONTROL OF BATCH PROGRAM EXECUTION WITH THE SAS SYSTEM

Transcription

1 CONTROL OF BATCH ROGRAM EXECUTION WITH THE SAS SYSTEM Tyler L. Witko, American Cyanamid Company Dr. Thomas R. Hoffman, American Cyanamid Company Abstract Information systems consisting of logically related data sets updated by batch s submitted independently by multiple users require a method of controlling batch job execution. Control is required to ensure updates to the data sets in proper order. By using the SAS System to write batch job execution requests to a sequential file and then processing the file with SAS Macro Language, a control module was developed that effects batch job scheduling, sequencing, and documenting of job execution histories for analysis. Although this module was developed for a pharmaceutical application on an IBM mainframe running under MVS, ~ could function for any type of information system on any platform that permits batch jobs to subm~ other batch jobs for execution. Introduction The need to sequence batch jobs that independently update logically dependent data sets is rather straightforward. For example, consider a simple database consisting of two data sets: SECS contains names of tests performed by a laboratory; RESULTS contains the actual data generated by laboratory testing. To process new laboratory test data into the RESULTS data set, the test name associated with the data must be validated by its existence in the SECS data set. Before successfully processing the results of a test not found in SECS into the RESULTS data set, the test name must be added to the SECS data set. In an environment with one person controlling data processing this task is simple: Update SECS, then update RESULTS. In a multi-user environment where each user has the authority to update SECS and RESULTS via batch s, coordinating these tasks becomes difficult. Jobs may be put on hold due to contention, executed redundantly, or may fail if executed out of proper sequence. Following is a description of an information system with over one hundred users, hundreds of data sets, and the module developed to coordinate batch execution. Database Structure The Clinical Information System, (THECIS), is a SAS System based clinical trials information system developed in-house at the Lederle Laboratories Division of American Cyanamid Company. It resides on an IBM 3084 running under MVS using the JES2 and UCC? batch control software. The database is logically structured into three levels: Metadata, Drug Level, and CRF Level. The Metadata level contains information about the data stored in the Drug and CRF levels, (data set names, structures, domains), system-wide format libraries, user definitions, etc. The Drug Level contains clinical trials information stored in a separate data library for each drug project. The data sets in each Drug Level library have structures identical to data sets in other Drug Level libraries. This similarity of structures facilitates ming for loading data and permits executing standard report s on any drug project. The CRF Level contains the same data stored in the Drug Level but in drug/protocol specific structures. The CRF Level data set structures are modeled after the Case Report Forms used to collect the data. All trials data are loaded and ed~ed in the CRF Level and moved into the Drug Level for reporting. Data is loaded from sequential data files and edited at the CRF Level using SAS/FSp and transaction data sets. All data loaded into the CRF Level and moved into Drug Level must conform to criteria specified in the Metadata Level. All updates to each levels' master data sets are achieved through overnight batch processing. Overnight updating permits unrestricted read access to all three levels during the day and shifts resource load. Batch Job Control Requirements like the simple example described above, ~ is cr~ical to update the three levels in a controlled sequence: A drug project must be defined at the Metadata Level before data for that project can be loaded into the CRF Level. Similarly, data must be loaded/updated at the CRF Level before it can be moved to Drug Level for reporting purposes. 933

2 Addftionally, report s run against the Drug Level should not execute prior to Drug Level update. Another requirement was the ability to schedule drug speckic updates for certain nights of the week. The option to override the schedule and execute an update during the day was required also. Finally, it was necessary to provide an on-line facility to review the current batch, prior night's activities, log files for failed jobs, and historical records of activity. Although JES2 and UCC7 have batch job control features, they were not robust enough to meet our requirements. While JES2 parameters can influence job sequencing, they do not provide the level of control required. As well, JES2 does not fulfill the drug project scheduling requirement. The UCC7 software provides greater control than JES2 but required maintenance by system mers and added complexity to the process of defining and/or scheduling drug projects. UCC7 does not have the abilfty to maintain historical data on job execution. Benefits of Batch Job Control Database updates occur in proper order, eliminating the need to re-execute jobs that ran out of sequence, reducing resource strain and update time. Faster response to failed jobs - job completion status is available on-line first thing in the morning - no need to wait for printed output. Less output - log files are only output" a job fails. Better control over database updating through scheduling - a priority drug project update will not be delayed due to another's processing. Resource planning through trend analysis of historical data. On-line review of job verify expected updates are scheduled. Batch Job Control Components To fulfill the requirements listed above, a change in the concept of independently submitting job. streams to JES2 for execution was needed. Batch jobs were assigned a 'class', according to the function of the job and targeted database level. The classes are described below: Class I Description 1 Metadata Level update - New drug projects, user definitions, format library updates, elc. 2 CRF Level update - Load new data, process transaction data sets from editing, initiate Drug Level updating, (submit class 3 jobs). 3 Drug Level update - move data from CRF to Drug Level 4 Report s figure 1 Batch Job Classification Instead of submitting JCL and statements to the JES2 job scheduler, job requests are written to a sequential data set called BJQUEUE throughout the day. The format of BJQUEUE is described below: Variable I Description QUEUE Queue number, (class) TIMEL Date/time logged USERIO Use rid logging request o Drug code or blank rotocol code or blank ROGRAM Batch name. STATUS Job status, (blank, HOLD, SCHEI figure 2 BJQUEUE file spechications The BJQUEUE file serves as a job and is processed by a group of interacting macros which logically process the job requests in a controlled sequence. A second sequential file, BJLOG, collects information about jobs executed and serves as a counter to determine when all jobs in a spechic class are cofllliete. The format of BJLOG is described below: Variable QUEUE TIMEL JUSERIO o I Description Queue number, (class) Oateltime logged Job statement use rid, (for output routing) Drug code or blank rotocol code or blank Batch name Oateltime completed Completion status:lblank or FAIL) ROGRAM TIMEC CSTATUS figure 3 BJLOG file spechications 934

3 The macros involved and the parameters passed between them are described below: bicontrl timel juserid d qsize cstatus force b'lnlt no arms bjaueue status bjqread no parms Ibil~ no~ms bjgen d p userid timel I qsize I Batch iob Drocess controller 0,1,2,3,4,5 datetime logged job statement userid, (output destination) drug code or. protocol code or. batch name expected number of jobs in completion status, (null, FAIL) force immediate execution erform module maintenance I ADDend records to BJOUEUE file 1,2,3,4 batch name iob status Extract job requests matching I current value of aueue I rocess BJLOG file Generate JCL and statements for batch jobs, submit for execution. drug code or. protocol code or. batch name TSO use rid that logged the job 1,2,3,4 datetime logged expected number of jobs in aueue figure 4 Macros and parameters Batch Job Control Lagle Throughout the day batch job requests are written as individual records to the BJOUEUE file by the BJOUEUE macro. At 7:00 each night, UCC7 submits a batch job to begin THECIS batch job execution. The initiating job invokes the BJCONTRL macro with & = O. When BJCONTRL executes with & = 0, it invokes the BJINIT macro. Although the BJINIT macro is physically the first macro run, it performs maintenance on the file resulting from the previous night's activity and its functions will be described last. At this point, it is only necessary to understand that it performs maintenance activities and returns control to BJCONTRL. BJCONTRL increments the value of & by 1 then invokes the BJOREAD macro. ' BJOREAD reads the sequential file BJQUEUE extracting records that match the value of &: stores the number of jobs in the BJCONTRL parameter qsize and returns control to BJCONTRL. If no records match the value of &, (&qsize = 0), the value of & is incremented by 1 and BJOREAD is invoked. This process continues until the value of & = 5. If records matching the value of & are found the macro BJGEN is invoked for each record returned. BJGEN generates the JCL and statements for each batch job and submits the job stream to the internal reader for execution. Each batch job contains four steps as follows: 1. THECIS 2. SUCCESS 3. FAIL erfoltll data updating Submit job to internal reader for execution which invokes BJCONTRL with CSTATUS = null. Submit job to internal reader for execution which invokes BJCONTRL with CSTATUS = FAIL. 4. JHS Output log to Job History SYstem for review. figure 5 JCL Steps Generated by BJGEN The ~~eps in figure 5 are executed on the following conditions: Step 1 always execules and performs system work. Step 2 only executes if step 1 does not fail. Steps 3 and 4 execute if step 1 fails. Note the job log is only stored if the THECIS step fails. The macro BJCONTRL is always invoked, (step 2 or 3). Separate jobs are executed to avoid JCL contention and ensure proper updating of the BJLOG file. BJCONTRL, with (& A= 0), invokes the BJLOG macro. BJLOG updates the BJLOG sequential file with information on the job executed in step 1. The BJLOG macro then counts the number of jobs in the BJLOG file, (NJOBS), with the current parameter, (OUEUE = &). If all jobs of the 935

4 current class have not completed, BJCONTRL is terminated. If all jobs in the current have completed, (NJOBS = &qsize), the parameter is again incremented and the process begins again forthe next job class, (see BJOREAD above). BJINIT is the first macro invoked each night to clean up the previous night's activhies. BJINIT generates a report detailing batch activity of the previous night from the information stored in BJLOG. Completed jobs found in the BJLOG ffie are compared with the job requests found in the BJOUEUE file, (keys = D,, ROGRAM, DATEL). Matches are wrhten to a SAS data set to maintain a history, successful jobs deleted from BJOUEUE, and BJLOG is re-inhialized to empty. This arrangement preserves jobs in the that were not executed due to system failure or not completed due to job failure. Refer to appendix I for a flowchart of the batch job control logic. Batch Job Scheduling When the STATUS field in the BJOUEUE file is 'SCHE', a job is only submitted when scheduled. Scheduling of batch jobs to execute drug specific updates on specific nights of the week is accomplished through the use of formats and the SAS weekday function. The DBA defined format $DLOAD. maps the zero filled concatenated value of D and into a string of five O's or 1's, representing the five workdays of the week. For example the string '01001' would indicate drug level updates be run on Tuesday and Friday. Non scheduled Ds, (OTHER), are mapped into a string of five 1's. Jobs not submhted remain in the until their scheduled day. A macro parameter FORCE is used to override the scheduling and execute the update immediately in the event a specific job must be run out of schedule. On-line Review Utilities were developed to permit on-line browsing of jobs in, (BJOUEUE), the BJLOG file of jobs run the prior night, or the job history data set. Job logs for failed jobs are available for review/printing through the Job History System utility. Conctuslons The batch job control module provides a means for achieving well defined management of database updating in an environment otherwise feft largely to chance. The module functions with no human intervention yet readily permits modifications to achieve flexibility. This logic can be deployed on any platform for any number of users. For business critical applications, this control translates into dollars earned by reducing the time required to complete data processing. Data processing charges are also reduced by eliminating redundant execution of jobs and printing of non-essential output. Resource tracking, capacity planning, and failure recovery are additional benefits realized from this module. Contact For addhional information, please contact: References TylerWHko American Cyanamid Company North Middletown Road 190/208C earl River, NY hone: SAS, and SAS/FS are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. IBM is a registered trademark or trademark of Intemational Business Machines Corporation. 936

5 Appendix I rogram Flowchart for Batch Job Control Module, 2 jobs in current class ry..::e::.s ~ o/objinit BJQUEUE BJLOG BJLOG qsize & >-_~ =5 No BJQUEUE No NOTE: %bjgen, SUCCESS, and FAIL submit jobs to internal reader 937