An Automation Procedure for Oracle Data Extraction and Insertion

Transcription

1 An Automation Procedure for Oracle Data Extraction and Insertion Shiqun S. Li, Smith Hanley, East Hanover, NJ David H. Wilcox, NYS Department of Health, Albany, NY ABSTRACT SAS software provides strong capability of extracting data from foreign databases such as Oracle tables. However, it would be very time-consuming if the database is very large. Even worse if the foreign databases cannot be accessed from the SAS programmers office, it would require them to make a trip to the client s site. In this paper, a real world application of downloading and uploading between SAS datasets and Oracle tables in a semi-automatic way will be presented. With such implementation, data retrieval becomes an easy routine task. Even for those with a very limited SAS knowledge, they can accomplish the data retrieval procedure. In this scheme, first, SAS Macro %Window was utilized to create an interface and collect the date range to extract the data; secondly, SAS/CONNECT and SAS/ACCESS were used to perform the extraction process; Thirdly, a registry dataset is generated and appended into the appropriate Oracle table as required by HIPAA law; Finally, a SAS macro will automatically the SAS log file and the extraction summary table to the Oracle DBA and SAS programmers for review. This paper is prepared for an intermediate and advanced audience. Key Words: SAS %Window, SAS/ACCESS, SAS/CONNECT, SAS Access, ORACLE Tables, Data Retrieval, MDS2.0 Data INTRODUCTION MDS2.0 (Minimum Data Set version 2.0) is a CMS (Center of Medicare and Medicaid Services) Oracle database. It collects assessments on a regular basis for every resident in a Medicare or Medicaid certified nursing home. The whole database has more than 40 tables with more than 800 variables. MDS assessments have become an important source for evaluating the quality of care and for reimbursing the cost of nursing home services. Over the past few years, extracting MDS data from MDS Oracle database to SAS data sets is one of the routine jobs for our projects. Due to the different location of the database, the restriction of remote access and the time-consuming extraction, we developed a semi-automatic approach so that the data retrieval procedure can be implemented with limited SAS programmers interference. In this paper, we will present our application in the following sequences: a) collect the parameters for data exporting; b) extract the data and convert them into SAS data sets; c) generate data summary tables; d) upload a summary table from SAS back to Oracle; e) automatically the SAS log and a summary table to Oracle DBA and SAS programmers for review and for archive. 1

2 PARAMETER COLLECTION Before the download procedure, some basic input information should be gathered. In our case, these items are: the data download time period (start date and end date), Oracle database user ID and the password (of course, you need password to access a security database). For simplicity, we use SAS macro %Window [1, 2] to create a user-friendly interface. The macro variables are then collected through this window. This input screen will look like this: The SAS codes that generate this window are: /*** Create Data Input Screen and collect the inputs ***/ %Window DatInput color=white rows =35 columns=80 'Welcome to MDS Extraction System' attr=highlight color=black "-- &sysday, &sysdate. --" color=black "Before start the procedure, please input the required parameters below." color=black 'Enter your ID: ' color=black _UserName_ 20 DISPLAY=YES REQUIRED=YES ATTR=UNDERLINE color=cyan 'Enter your password: ' color=black _Pswd_ 20 DISPLAY=No REQUIRED=YES ATTR=UNDERLINE color=cyan 2

3 'Enter START DATE of the download period: (MM/DD/YYYY) ' color=black _STARTDATE_ 12 Display=YES REQUIRED=YES ATTR=UNDERLINE color=cyan 'Enter END DATE of the download period: (MM/DD/YYYY) ' color=black _ENDDATE_ 12 Display=YES REQUIRED=YES ATTR=UNDERLINE color=cyan 'Press Enter to continue.' color=black "To exit the system, type " color=black "'exit'" ATTR=UNDERLINE color=black " at COMMAND line" color=black "You are using SAS v&sysver. " color=black ; %Display DatInput; In order to make sure the data entered is as desired, we provide a confirmation screen to give users a chance to review their entries. If there is a need for some modifications, it will allow the users to go back to the input screen for changes. After the entries are confirmed, the program will call the downloading macros and start the extraction process. The confirmation interface was designed as follow: 3

4 The corresponding SAS codes for this screen are listed below: *** Display the information collected and to make sure the information is accurate; %let _confirm_=; %window Confirm color=white rows =35 columns=80 'Welcome to MDS Extraction System' attr=highlight color=black "-- &sysday, &sysdate. --" color=black "Please confirm the information you entered are correct." color=black "Your ID: " color=black "&_UserName_" ATTR=UNDERLINE color=blue 'Your password: ' color=black '(Not displayed)' ATTR=UNDERLINE color=blue 'START DATE of the download period (MM/DD/YYYY): ' color=black "&_STARTDATE_." ATTR=UNDERLINE color=blue 'END DATE of the download period (MM/DD/YYYY): ' color=black "&_ENDDATE_." ATTR=UNDERLINE color=blue 'Is the information above correct and ready to download the data?' color=black '1 -- ' color=blue +1 'Yes' attr=underline color=blue +2 '(Will start extracting right after this screen)' color=blue '0 -- ' color=black +2 'No' attr=underline color=black +2 '(Will go back to the previous screen for changing the inputs)' color=black 'Q -- ' color=red 'Quit' attr=underline color=red +2 '(Will quit the extraction system)' color=red "Please select the appropriate choice:" color=black +2 _Confirm_ 2 attr=underline color=blue 'Press Enter to continue.' color=black ; %Display Confirm; This is a simplified set of parameter collection screens. In reality, we collect several more parameters like account ID and account password etc. SAS macro %Window does provide an easy technique for inputting the necessary variables. However, SAS SCL can create better looking and more advanced interfaces. But it is out of the scope of our topic, it will not addressed here. DATA EXTRACTION Before downloading the data, we need to create locations for storing the extracted data and the summary tables. SAS statements below will make the sub-directories for our needs: Options noxwait; X mkdir E:\MDS\MySubDir\&sysdate. ; 4

5 With all the parameters collected and the data storage directories created, we may now start to import the data. There are several methods to access Oracle tables. The way with the simplest syntax is to utilize Oracle LIBNAME engine [3]. In order to be able to use the following method, SAS/ACCESS [4, 5] must be licensed at your site. The LIBNAME statement can be specified as follows: LIBNAME MyDb ORACLE User=&_UserName_. ORAPW=&_Pswd_. PATH=MyOraPath SCHEMA=MyOraSchema; Where MyOraPath is the name of the ORACLE database; MyOraSchema is the name of the ORACLE schema. After this LIBNAME linkage is set up, the dataset names and variable names of all Oracle tables associated with the specified path and schema can be listed by: Proc contents data=mydb._all_; Run; As a matter of fact, all Oracle tables under this LIBNAME can be read by SAS now, and therefore can be copied into SAS system. In our real situation, we use the following SAS/SQL sample code to extract the MDS Oracle tables: Proc SQL; Connect to Oracle as sasquery (user=&_username_. Orapw=&_Pswd_. ); Create table &Dir.&TableName. As Select * from Connection to sasquery (Select * from &TableName. &WithCondition.); Disconnect from sasquery; Quit; Since there are 40 tables to be downloaded and most tables are rather large, this part is the most time-consuming. It would take 3 to 4 hours to retrieve the tables of our interest. MANIPULATING EXTRACTED SAS DATASETS A SAS data set converted from an Oracle table usually requires some minor changes. For instance, a date variable from Oracle is typically of DATETIME20. format. We should standardize it as DATE9. format using SAS DATEPART function. The numerical variables with single digit values, such as gender variable, are re-defined to be length=3. Finally, as required by our project, we merge some specific SAS datasets together. All those steps are standard SAS data manipulation. Anyone who is interested in our approach can us for the detailed SAS macros. 5

6 SUMMARIZING THE EXTRACTION DATASETS It is always a good practice to generate a summary report after each extraction. This will benefit everyone when there is a need for HIPAA auditing or data download questions. There are two summary files we generate. The first one is to summarize the tables we download, with the table names, number of records and variables of each table. A small macro below, alone with CALL EXECUTE, will do this job: %macro ObsVar(dsn=); %let dsid = %sysfunc(open(&dsn)); %if &dsid %then %do; %let nobs =%sysfunc(attrn(&dsid,nobs)); %let nvars=%sysfunc(attrn(&dsid,nvars)); %let rc = %sysfunc(close(&dsid)); dsn="&dsn."; NOBS=&Nobs. ; NVARs=&NVars.; output; %end; %else %do; *** If the dataset does not exist; %put Open for data set &dsn. failed - %sysfunc(sysmsg()); dsn="&dsn."; NOBS=.; NVARS=.; output; %end; %mend ObsVar; The second summary file we produce is required by HIPAA regulation (Health Insurance Portability and Accountability Act). It is mandatory to register a log record to a CMS HIPAA database whenever a nursing home resident record is extracted. This data set should include the basic resident information like resident ID, SSN, assessment date, extraction date and project DUA (Data Use Agreement number). Based on the information we imported, it is fairly easy to generate this HIPAA log file. UPLOADING HIPAA REGISTER After the HIPPA log file is generated, it is necessary to upload this SAS data set back to HIPAA registers in CMS Oracle database. The SAS code below is used to insert the HIPAA log file into the designated Oracle table: LIBNAME pad Oracle User=&user Orapw=&orapw Path=&path schema=&schema2; PROC SQL; Insert into pad.pad_dtl( State_ID, Res_Int_ID, SYS_IND, SYS_DT, EXTR_DT, DUA_NBR, ASMT_SUBM_DT, ASMT_REF_DT, LNAME, FNAME, GENDER, DOB, SSN) Select State_ID, Res_Int_ID, SYS_IND, SYS_DT, EXTR_DT, DUA_NBR, ASMT_SUBM_DT, ASMT_REF_DT, LNAME, FNAME, GENDER, DOB, SSN From MDS_SUMM.HIPAA_Log_&YMD. (firstobs=&_from. obs=&_to. ) ; quit; 6

7 ING SAS LOG FILE The design of our data extraction process is for people with little SAS experience. Therefore, after downloading, uploading and summarizing the data, it is important to thoroughly scrutinize the SAS log file by an experienced SAS programmer. It is also necessary to carefully review the summary file of the data sets we have extracted. To accomplish these purposes, we use SAS Access method [6, 7] to automatically send the SAS log file and the extraction summary file to a specified SAS programmer and a MDS database coordinator. The programmer can then review the SAS log from any location. SAS macro as shown below can achieve this task: filename To ATTACH=( "E:\MDS\MDS_Extraction_&YYMMDD..log", "E:\MDS\MDS_Summary_&YYMMDD..txt") SYS=VIM ID="Stan Li/PH/NovXXX" PW="&My PassWord." ; data _null_; file To ; put '!EM_TO!' "shiqun@gmail.com" ; put '!EM_CC! DBA_eFacility@eFacility.com'; put '!EM_SUBJECT! MDS Data Extraction Reports for your Review/Record' ; put 'Hi,'/; put 'Attached are the reports from MDS Data Extraction System.'; put Please take a moment to review those reports attached. ; put 'Feel free to call me if there is any question. Thank you.'; put //; put 'Best Regards,'; put /; put &_UserName_. ; put '!EM_SEND!'; *** send it now; put '!EM_NEWMSG!'; put '!EM_ABORT!'; *** to avoid from send it once more; run; Here we assume the SAS log file is saved to a file using PROC PRINTTO. It should be noted that this example is based on Lotus Notes system on Window XP platform. For different systems, some parameters may be different. For instance, in Microsoft Outlook, you should define: SYS=MAPI ID="sli@efacility.com" PW=" PassWord". CONCLUSION Our data extraction application is easy to use. There is not need to do any alteration on the SAS codes when executing the download procedure. The necessary input parameters are collected from interface screens. Throughout the entire procedure, SAS knowledge is not required for users to implement it. This is cost-effective and may avoid some travels for SAS programmers when the database is located off-site, such as our case. Our codes are designed for extracting nursing home MDS2.0 data. But the scheme can be easily modified to fit your specific business need. 7

8 REFERENCES 1. Kay Alden, SAS Best Kept Secret: Macro Windows for Applications Development. SUGI 25, 2. Linda C. Gau, Using SAS Software Window to Dynamically Manage Your Routine Programs. SUGI 24, 3. David D. Chapman, Using the ORACLE LIBNAME Engine to Reduce the Time It Takes to Extract Data From an ORACLE Database. NESUG 2006, 4. SAS/ACCESS: Interface to Relational Databases. SAS Reference, 5. Frederick Pratter, Database Access Using the SAS System. SUGI 26, 6. Wenjie Wang and Simon Lin, Using SAS to Send Bulk s With Attachments. ParmaSug 2007, 7. David H. Johnson, E mail from your SAS Session: How and Why. SUGI 31, CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the authors at: Shiqun (Stan) Li Smith Hanley, East Hanover, NJ (908) shiqun@gmail.com David H. Wilcox NYS Department of Health, Albany, NY (518) dhw01@health.state.ny.us SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 8