Using SAS 9.4M5 and the Varchar Data Type to Manage Text Strings Exceeding 32kb
|
|
- Christal Edwards
- 5 years ago
- Views:
Transcription
1 ABSTRACT Paper Using SAS 9.4M5 and the Varchar Data Type to Manage Text Strings Exceeding 32kb John Schmitz, Luminare Data LLC Database systems support text fields much longer than the 32kb limit traditionally supported by SAS. These fields can be captured as substrings using SAS and explicit pass-through logic. However, managing the resulting substrings is often challenging. This paper will show how the varchar data type, introduced in 9.4M5, can provide improved functionality. The paper solves common management and storage issues associated with the extraction of longer strings, using varchar field type and the V9 engine. INTRODUCTION Strings longer than 32kb (32767) characters require additional steps to fully extract into a SAS readable format. Although these strings are commonly supported in most database management systems (DBMS), they are not fully supported in SAS DATA steps or in the SAS/ACCESS engines that facilitate the transfer of data between the DBMS and SAS. Beginning with version 9.4M5, SAS has introduced features that will aid programmers when managing these longer string variables. However, these new capabilities are not a complete solution to the challenges such data offer. This paper provides suggestions to simplify text processing in SAS when longer strings are encountered. The paper will first review the VARCHAR data type that is new in 9.4 M5. The paper will briefly touch on the traditional approach to handling long strings for systems prior to 9.4M5, and then present some new methods that can be used with the VARCHAR feature in M5. This paper assumes the reader is familiar with general SAS coding including the SQL procedure and SQL pass-through logic. The paper provides only a brief outline of the process for extracting these longer strings into SAS, so that more attention may be focused on the management of these longer strings once they have been transferred to SAS. The code samples shown use the SAS/ACCESS for SQL Server engine, but the same process would apply for other engines including ODBC and OLEDB. THE VARCHAR DATA TYPE SAS has introduced a new VARCHAR data type with SAS 9.4 M5. This new data type is included to support SAS VIYA and the new HDAT data type used within the CAS Engine. However, the core capabilities of VARCHAR are included in the DATA step as part of the BASE SAS license, whether you have VIYA and CAS in your SAS environment or not. A VARCHAR data type is declared using the LENGTH statement, using the keyword VARCHAR: LENGTH longstr VARCHAR(n); where LONGSTR is the user-defined variable name and n is a numeric value that defines the maximum length that the variable string LONGSTR may assume. The maximum length of a VARCHAR is or 536,870,911 compared to the maximum length of or 32,767 for a fixed length string. Unlike SAS standard character variables, a VARACHAR cannot be declared with a FORMAT statement. The SAS V9 engine does not support VARCHAR fields as a storage type. Therefore, the VARCHAR field will be available within your DATA step but will not be outputted as a VARCHAR when a V9 dataset is used as the output table. Rather, for the V9 engine, the fields will be converted to a fixed length field of no more than and stored as a CHAR variable. Any characters in the string beyond the field length will be truncated. As an example, consider a simple case: %put SAS Version: &SYSVLONG; options msglevel=i; 1
2 data sample1; longstr = 'ABC'; title 'Contents of SAMPLE1 Dataset'; proc contents data=sample1; proc print data=sample1; In this case, LONGSTR is a VARCHAR with a maximum length of 160,000 characters. The log for this example shows a few items worth noting, namely a report of the SAS version used: %put SAS Version: &SYSVLONG; SAS Version: M5P and a message regarding the VARCHAR datatype and the V9 engine: NOTE: VARCHAR data type is not supported by the V9 engine. Variable longstr has been converted to CHAR data type. The CONTENTS procedure output confirms the field conversion indicated in the log:: Alphabetic List of Variables and Attributes # Variable Type Len 1 longstr Char The PRINT procedure shows the data remain intact in output, since the string was short enough to fit within the field length. Contents of SAMPLE1 Dataset Obs longstr 1 ABC PROCESSING LONGER STRINGS PRIOR TO 9.4 M5 The common approach to extracting longer strings to SAS would involve: Leveraging SQL pass-through to substring fields into string lengths no more than 32kb. Transferring the substring segments to SAS as separate strings. Processing data within the DATA step using each of the string subsets, often within a loop. Some business requirements may require pre-processing on the substrings to create more natural break points. This is done so that the desired text is not split across 2 substrings. Depending on the users desired outcome and the number of substrings involved, this string processing across the substring segments can become a complex process. Readers interested in a more thorough presentation on the extraction and processing of such strings should review the paper by Schmitz (2017) included in the References section. 2
3 Assume we have a SQL data table with XML strings included as a VARCHAR field. For this example, we will assume a maximum string length of 160,000 characters so 5 substrings of 32,767 characters each will be adequate to extract the entire field. SQL pass-through code can be used to divide the entry into 5 substrings and return the string as multiple substrings: proc sql; connect to sqlsvr as src (dbmax_text=32767 dsn=lumin_dev authdomain=sqlauth); create view XML_SampleData_vw as select * from connection to src (select tableid, substring(xmlresponse, 1, 32767) as XMLResponse1, substring(xmlresponse, 32768, 65534) as XMLResponse2, substring(xmlresponse, 65535, 98301) as XMLResponse3, substring(xmlresponse, 98302,131068) as XMLResponse4, substring(xmlresponse,131069,163835) as XMLResponse5 from dbo.xml_sampledata); disconnect from src; quit; This will generate a DATA step view with a TableID (table key) and 5 substrings with subsets of the XMLResponse field. Note that DBMAX_TEXT option is used in the connect statement. This setting will override the default maximum length for character data used by the ODBC drivers. This setting commonly defaults to 1000 characters. Continuing with this example, assume we are looking for an ID value that appears in the text immediate after a specific UUID value. The DATA view can serve as input to a SAS DATA step for the additional string processing requirments. In this example, the code searches for a particular UUID value within the substring text then captures the next 12 characters following the identified string. In a pre-varchar process, the basic data steps would appear as: data XML_Sample; ** CREATE LOOP TO SEARCH ACROSS SUBSTRINGS WITH FIND **; pos=0; do i = 1 to 5 until (pos > 0); pos = find (XMLS(i),"DFE24CAA-04EF-4F81-86D8-9F3BABBA616B"); ** IF FOUND, CAPTURE NEXT 12 CHARACTERS FROM STRING **; length IDString $12; if pos > 0 then IDString = substr (XMLS(i),pos+ length("dfe24caa-04ef-4f81-86d8-9f3babba616b"),12); 3
4 This code loads the view created by the SQL pass-through above and searches across each substring for the position of the desired UUID code. Here, a SET statement is used and an ARRAY is declared to facilitate a loop across all the strings. A subsequent DO UNTIL loop is used to search each string for the desired code. Once found, the position is recorded relative to that particular substring and processing is passed to a subsequent SUBSTR command to capture the next 12 characters s IDString. Of course, this approach has 1 critical limitation in that it cannot find the desired string if it is split across two substrings. It would require additional coding to concatenate strings across the break points to ensure no values are missed due to this limitation. This approach requires creating natural breaks in the string subsets or pre/post appending data from an adjoining string. Either approach would typically require that the substring extracted from the DBMS be less than the maximum characters to allow room for this additional string manipulation process. Since the focus of this paper is upon improving this process with VARCHAR, there is minimal benefit in developing that logic here. Again, readers who are interested in more details on this approach are directed to Schmitz (2017). MODIFYING THE APPROACH FOR M5 AND VARCHAR Because the M5 code additions do not alter the SAS/ACCESS engines to allow longer strings, it is still necessary to execute a SQL pass-through approach to subset and extract the data. However, the VARCHAR field type does allow the user to reconstruct the full string inside the DATA step, up to the 536,870,911 character limit. In most cases, this approach will allow the user to bypass many of the substring limitations raised above. In M5, an improved DATA step using the VARCHAR type can be constructed as: data XML_Sample2; ** FIND AND SUBSTR WILL NOW WORK OVER ENTIRE VARCHAR FIELD **; pos = find (longstr,"dfe24caa-04ef-4f81-86d8-9f3babba616b"); ** IF FOUND, CAPTURE NEXT 12 CHARACTERS FROM STRING **; length IDString $12; if pos > 0 then IDString = substr (longstr,pos+ length("dfe24caa-04ef-4f81-86d8-9f3babba616b"),12); The SET and ARRAY statements are retained as before. A LENGTH statement is used to declare a VARCHAR variable and a DO loop is used to concatenate each substring to the declared VARCHAR field. Once populated, the FIND and SUBSTR functions can search the entire VARCHAR field to obtain the desired outcome. On the surface, this new approach may not appear any simpler. However, the previous approach bypasses any logic to manage cases where breaks within the substrings will result in inaccurate and/or incomplete results. In the latter case, there are no substring sections, so the logic can find the patterns even when they appear across substrings. Furthermore, there is no need to loop over substrings each time the user processes against the string data since the single VARCHAR holds the entire field. 4
5 KEY LIMITATIONS TO THE VARCHAR PROCESS Although the VARCHAR type provides a much easier process for searching and processing on these longer strings, several limitations with the VARCHAR data type should be addressed. One of the key limitations is saving the file since the VARCHAR is not supported in V9 engine. One option is to drop the field but retain the substrings captured from the SQL pass-through. The VARCHAR can be readily reconstructed from the substrings as desired. data XML_Save; ** DROP VARCHAR BEFORE SAVE **; drop longstr; data XML_Load; set XML_Save; ** RECREATE AND POPULATE VARCHAR FIELD **; An alternative is to write the VARCHAR as a text file. As an example: filename outf1 "~/Documents/outf1.txt"; data XML_Sample3; file outf1 lrecl=160000; put longstr; 5
6 From the log: NOTE: 2 records were written to the file OUTF1. The minimum record length was The maximum record length was The sample code provided uses a loop concatenate multiple strings together. This is important since using a single CAT function to join all fields together will fail. Consider the code: longstr = cat(xmls(1),xmls(2),xmls(3),xmls(4),xmls(5)); This code produces an error due to buffer limits in the string function: WARNING: In a call to the CAT function, the buffer allocated for the result was not long enough to contain the concatenation of all the arguments. The correct result would contain characters, but the actual result might either be truncated to character(s) or be completely blank, depending on the calling environment. The following note indicates the left-most argument that caused truncation. Buffer errors are not uncommon using VARCHAR when string functions must buffer too many characters. The SUBSTR function works as expected when extracting from a VARCHAR to a CHAR for results up to the maximum CHAR field length of characters. Consider the code: data XML_Sample5b; ** SUBSTR WILL WORK OVER ENTIRE VARCHAR FIELD **; length FindString varchar(80000); FindString = substr (longstr,500,80000); FindString = substr (longstr,32768,80000); FindString = substr (longstr,125000,80000); 6
7 title Summary Output from XML_Sample5b; proc print data=xml_sample5b; var FindString FindLen; format FindString $10.; The PRINT procedure shows that strings were returned as expected: Summary Output from XML_Sample5 Obs FindString FindLen 1 //data.lac </total_p /zip_code> g steps to mnid=" responds 2491 A similar example can be constructed using a VARCHAR rather than the CHAR field to capture the result of the substring. data XML_Sample5b; ** SUBSTR WILL WORK OVER ENTIRE VARCHAR FIELD **; length FindString varchar(80000); FindString = substr (longstr,500,80000); FindString = substr (longstr,32768,80000); FindString = substr (longstr,75000,80000); title Summary Output from XML_Sample5b; proc print data=xml_sample5b; var FindString FindLen; format FindString $10.; 7
8 Again, the PRINT procedure confirms the desired substrings were returned. Summary Output from XML_Sample5b Obs FindString FindLen 1 //data.lac </total_p tion="172" g steps to mnid=" Config><vi However, the user should note that in either case, the final string will be truncated if the SUBSTRING range extends beyond the MAX length defined for the VARCHAR source string. In the above example, use of a start position of 130,000 and string length of 32767: FindString = substr (longstr,130000,32767); results in a NOTE in the log and a blank value for the returned substring since the command (130, ) exceeds the maximum defined VARCHAR length of 160,000. NOTE: Argument 3 to function SUBSTR('<?xml versio'[12 of characters shown],130000,32767) at line 50 column 15 is invalid. CONCLUSION The new VARCHAR data type in SAS 9.4M5 can simplify string extraction and manipulation within the DATA step for data sources that have strings beyond the traditional 32kb limit for as CHAR variables. The VARCHAR will allow the user to scan, search and substring across the entire long string, up to a maximum length exceeding 500 Million characters. The process is not without limitations. The VARCHAR does not alter the SAS/ACCESS engine limit on a 32kb maximum string returned from a DBMS. Furthermore, the SAS V9 engine will not save the VARCHAR data type; although alternative methods can be used to save the data for later use. Also, users must exercise caution when using these longer strings since limitations on string functions can cause unexpected results. Users who need to process longer string data within the SAS DATA step can benefit from the new VARCHAR data type, even when VIYA and CAS are not available as part of the SAS installation. REFERENCES Schmitz, John, Extraction and Use of Text Strings With SAS When Source Exceeds the 32K String Length Limit. Midwest SAS Users Group Conference Proceedings, St Louis MO. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: John Schmitz Luminare Data LLC John.Schmitz@LuminareData.com SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 8
DATA Step in SAS Viya : Essential New Features
Paper SAS118-2017 DATA Step in SAS Viya : Essential New Features Jason Secosky, SAS Institute Inc., Cary, NC ABSTRACT The is the familiar and powerful data processing language in SAS and now SAS Viya.
More informationThe Future of Transpose: How SAS Is Rebuilding Its Foundation by Making What Is Old New Again
Paper 701-2017 The Future of Transpose: How SAS Is Rebuilding Its Foundation by Making What Is Old New Again Scott Mebust, SAS Institute Inc., Cary, NC ABSTRACT As computer technology advances, SAS continually
More informationSAS 9 Programming Enhancements Marje Fecht, Prowerk Consulting Ltd Mississauga, Ontario, Canada
SAS 9 Programming Enhancements Marje Fecht, Prowerk Consulting Ltd Mississauga, Ontario, Canada ABSTRACT Performance improvements are the well-publicized enhancement to SAS 9, but what else has changed
More informationSAS Viya 3.1 FAQ for Processing UTF-8 Data
SAS Viya 3.1 FAQ for Processing UTF-8 Data Troubleshooting Tips for Processing UTF-8 Data (Existing SAS Code) What Is the Encoding of My Data Set? PROC CONTENTS displays information about the data set
More informationPaper Haven't I Seen You Before? An Application of DATA Step HASH for Efficient Complex Event Associations. John Schmitz, Luminare Data LLC
Paper 1331-2017 Haven't I Seen You Before? An Application of DATA Step HASH for Efficient Complex Event Associations ABSTRACT John Schmitz, Luminare Data LLC Data processing can sometimes require complex
More informationTaming a Spreadsheet Importation Monster
SESUG 2013 Paper BtB-10 Taming a Spreadsheet Importation Monster Nat Wooding, J. Sargeant Reynolds Community College ABSTRACT As many programmers have learned to their chagrin, it can be easy to read Excel
More informationET01. LIBNAME libref <engine-name> <physical-file-name> <libname-options>; <SAS Code> LIBNAME libref CLEAR;
ET01 Demystifying the SAS Excel LIBNAME Engine - A Practical Guide Paul A. Choate, California State Developmental Services Carol A. Martell, UNC Highway Safety Research Center ABSTRACT This paper is a
More informationHow to Incorporate Old SAS Data into a New DATA Step, or What is S-M-U?
Paper 54-25 How to Incorporate Old SAS Data into a New DATA Step, or What is S-M-U? Andrew T. Kuligowski Nielsen Media Research Abstract / Introduction S-M-U. Some people will see these three letters and
More informationTips for Mastering Relational Databases Using SAS/ACCESS
Tips for Mastering Relational Databases Using SAS/ACCESS SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other
More informationDBLOAD Procedure Reference
131 CHAPTER 10 DBLOAD Procedure Reference Introduction 131 Naming Limits in the DBLOAD Procedure 131 Case Sensitivity in the DBLOAD Procedure 132 DBLOAD Procedure 132 133 PROC DBLOAD Statement Options
More informationTweaking your tables: Suppressing superfluous subtotals in PROC TABULATE
ABSTRACT Tweaking your tables: Suppressing superfluous subtotals in PROC TABULATE Steve Cavill, NSW Bureau of Crime Statistics and Research, Sydney, Australia PROC TABULATE is a great tool for generating
More informationLet the CAT Out of the Bag: String Concatenation in SAS 9
Let the CAT Out of the Bag: String Concatenation in SAS 9 Joshua M. Horstman, Nested Loop Consulting, Indianapolis, IN, USA ABSTRACT Are you still using TRIM, LEFT, and vertical bar operators to concatenate
More informationChecking for Duplicates Wendi L. Wright
Checking for Duplicates Wendi L. Wright ABSTRACT This introductory level paper demonstrates a quick way to find duplicates in a dataset (with both simple and complex keys). It discusses what to do when
More informationCMISS the SAS Function You May Have Been MISSING Mira Shapiro, Analytic Designers LLC, Bethesda, MD
ABSTRACT SESUG 2016 - RV-201 CMISS the SAS Function You May Have Been MISSING Mira Shapiro, Analytic Designers LLC, Bethesda, MD Those of us who have been using SAS for more than a few years often rely
More informationPagina 1 di 5 13.1.4. INSERT Syntax 13.1.4.1. INSERT... SELECT Syntax 13.1.4.2. INSERT DELAYED Syntax INSERT [LOW_PRIORITY DELAYED HIGH_PRIORITY] [IGNORE] [INTO] tbl_name [(col_name,...)] VALUES ({expr
More informationQUEST Procedure Reference
111 CHAPTER 9 QUEST Procedure Reference Introduction 111 QUEST Procedure Syntax 111 Description 112 PROC QUEST Statement Options 112 Procedure Statements 112 SYSTEM 2000 Statement 114 ECHO ON and ECHO
More informationA Practical Introduction to SAS Data Integration Studio
ABSTRACT A Practical Introduction to SAS Data Integration Studio Erik Larsen, Independent Consultant, Charleston, SC Frank Ferriola, Financial Risk Group, Cary, NC A useful and often overlooked tool which
More informationSAS/ACCESS Data Set Options
43 CHAPTER 4 SAS/ACCESS Data Set Options Introduction 43 SAS/ACCESS Data Set Options 43 Introduction This chapter describes the SAS/ACCESS options that you can specify on a SAS data set in the form SAS/ACCESS-libref.dbms_table_name.
More informationSame Data Different Attributes: Cloning Issues with Data Sets Brian Varney, Experis Business Analytics, Portage, MI
Paper BB-02-2013 Same Data Different Attributes: Cloning Issues with Data Sets Brian Varney, Experis Business Analytics, Portage, MI ABSTRACT When dealing with data from multiple or unstructured data sources,
More informationWorking with Big Data in SAS
ABSTRACT Paper SAS2160-2018 Working with Big Data in SAS Mark L. Jordan, SAS Institute Inc. This paper demonstrates challenges and solutions when using SAS to process large data sets. Learn how to do the
More informationText Generational Data Sets (Text GDS)
Paper 274-2017 Text Generational Data Sets (Text GDS) Dr. Kannan Deivasigamani HSBC ABSTRACT This paper offers a way to fill the void that SAS currently has with respect to the missing feature in the language,
More informationAutomatic Indicators for Dummies: A macro for generating dummy indicators from category type variables
MWSUG 2018 - Paper AA-29 Automatic Indicators for Dummies: A macro for generating dummy indicators from category type variables Matthew Bates, Affusion Consulting, Columbus, OH ABSTRACT Dummy Indicators
More informationSQL: Concepts. Todd Bacastow IST 210: Organization of Data 2/17/ IST 210
SQL: Concepts Todd Bacastow IST 210: Organization of Data 2/17/2004 1 Design questions How many entities are there? What are the major entities? What are the attributes of each entity? Is there a unique
More information29 Shades of Missing
SESUG 2015 ABSTRACT Paper CC106 29 Shades of Missing Darryl Putnam, Pinnacle Solutions, LLC Missing values can have many flavors of missingness in your data and understanding these flavors of missingness
More informationSquare Peg, Square Hole Getting Tables to Fit on Slides in the ODS Destination for PowerPoint
PharmaSUG 2018 - Paper DV-01 Square Peg, Square Hole Getting Tables to Fit on Slides in the ODS Destination for PowerPoint Jane Eslinger, SAS Institute Inc. ABSTRACT An output table is a square. A slide
More informationLeverage custom geographical polygons in SAS Visual Analytics
ABSTRACT Paper SAS1732-2018 Leverage custom geographical polygons in SAS Visual Analytics Falko Schulz, SAS Institute Inc., Brisbane, Australia Discover how you can explore geographical maps using your
More informationEssentials of PDV: Directing the Aim to Understanding the DATA Step! Arthur Xuejun Li, City of Hope National Medical Center, Duarte, CA
PharmaSUG 2013 - Paper TF17 Essentials of PDV: Directing the Aim to Understanding the DATA Step! Arthur Xuejun Li, City of Hope National Medical Center, Duarte, CA ABSTRACT Beginning programmers often
More informationIntroduction. Understanding SAS/ACCESS Descriptor Files. CHAPTER 3 Defining SAS/ACCESS Descriptor Files
15 CHAPTER 3 Defining SAS/ACCESS Descriptor Files Introduction 15 Understanding SAS/ACCESS Descriptor Files 15 Creating SAS/ACCESS Descriptor Files 16 The ACCESS Procedure 16 Creating Access Descriptors
More informationExporting Variable Labels as Column Headers in Excel using SAS Chaitanya Chowdagam, MaxisIT Inc., Metuchen, NJ
Paper 74924-2011 Exporting Variable Labels as Column Headers in Excel using SAS Chaitanya Chowdagam, MaxisIT Inc., Metuchen, NJ ABSTRACT Excel output is the desired format for most of the ad-hoc reports
More information9 Ways to Join Two Datasets David Franklin, Independent Consultant, New Hampshire, USA
9 Ways to Join Two Datasets David Franklin, Independent Consultant, New Hampshire, USA ABSTRACT Joining or merging data is one of the fundamental actions carried out when manipulating data to bring it
More informationWrangling Your Data into Shape for In-Memory Analytics
ABSTRACT Paper SAS567-2017 Wrangling Your Data into Shape for In-Memory Analytics Gary Mehler, SAS institute Inc., Cary, NC High-quality analytics works best with the best-quality data. Preparing your
More informationArthur L. Carpenter California Occidental Consultants, Oceanside, California
Paper 028-30 Storing and Using a List of Values in a Macro Variable Arthur L. Carpenter California Occidental Consultants, Oceanside, California ABSTRACT When using the macro language it is not at all
More informationImporting CSV Data to All Character Variables Arthur L. Carpenter California Occidental Consultants, Anchorage, AK
PharmaSUG 2017 QT02 Importing CSV Data to All Character Variables Arthur L. Carpenter California Occidental Consultants, Anchorage, AK ABSTRACT Have you ever needed to import data from a CSV file and found
More informationSAS Drug Development Program Portability
PharmaSUG2011 Paper SAS-AD03 SAS Drug Development Program Portability Ben Bocchicchio, SAS Institute, Cary NC, US Nancy Cole, SAS Institute, Cary NC, US ABSTRACT A Roadmap showing how SAS code developed
More informationThe Art of Defensive Programming: Coping with Unseen Data
INTRODUCTION Paper 1791-2018 The Art of Defensive Programming: Coping with Unseen Data Philip R Holland, Holland Numerics Limited, United Kingdom This paper discusses how you cope with the following data
More informationData Manipulation with SQL Mara Werner, HHS/OIG, Chicago, IL
Paper TS05-2011 Data Manipulation with SQL Mara Werner, HHS/OIG, Chicago, IL Abstract SQL was developed to pull together information from several different data tables - use this to your advantage as you
More informationINTRODUCTION TO SAS HOW SAS WORKS READING RAW DATA INTO SAS
TO SAS NEED FOR SAS WHO USES SAS WHAT IS SAS? OVERVIEW OF BASE SAS SOFTWARE DATA MANAGEMENT FACILITY STRUCTURE OF SAS DATASET SAS PROGRAM PROGRAMMING LANGUAGE ELEMENTS OF THE SAS LANGUAGE RULES FOR SAS
More informationUsing SAS Enterprise Guide to Coax Your Excel Data In To SAS
Paper IT-01 Using SAS Enterprise Guide to Coax Your Excel Data In To SAS Mira Shapiro, Analytic Designers LLC, Bethesda, MD ABSTRACT Kirk Paul Lafler, Software Intelligence Corporation, Spring Valley,
More informationDictionary.coumns is your friend while appending or moving data
ABSTRACT SESUG Paper CC-41-2017 Dictionary.coumns is your friend while appending or moving data Kiran Venna, Dataspace Inc. Dictionary.columns is a dictionary table, which gives metadata information of
More informationSeamless Dynamic Web (and Smart Device!) Reporting with SAS D.J. Penix, Pinnacle Solutions, Indianapolis, IN
Paper RIV05 Seamless Dynamic Web (and Smart Device!) Reporting with SAS D.J. Penix, Pinnacle Solutions, Indianapolis, IN ABSTRACT The SAS Business Intelligence platform provides a wide variety of reporting
More informationChapter 1 SQL and Data
Chapter 1 SQL and Data What is SQL? Structured Query Language An industry-standard language used to access & manipulate data stored in a relational database E. F. Codd, 1970 s IBM 2 What is Oracle? A relational
More informationProgram Validation: Logging the Log
Program Validation: Logging the Log Adel Fahmy, Symbiance Inc., Princeton, NJ ABSTRACT Program Validation includes checking both program Log and Logic. The program Log should be clear of any system Error/Warning
More informationMerging Data Eight Different Ways
Paper 197-2009 Merging Data Eight Different Ways David Franklin, Independent Consultant, New Hampshire, USA ABSTRACT Merging data is a fundamental function carried out when manipulating data to bring it
More informationReading in Data Directly from Microsoft Word Questionnaire Forms
Paper 1401-2014 Reading in Data Directly from Microsoft Word Questionnaire Forms Sijian Zhang, VA Pittsburgh Healthcare System ABSTRACT If someone comes to you with hundreds of questionnaire forms in Microsoft
More informationKey Reasons for SAS Data Set Size Difference by SAS Grid Migration Prasoon Sangwan, Piyush Singh, Tanuj Gupta TATA Consultancy Services Ltd.
SCSUG-2016 Key Reasons for SAS Data Set Size Difference by SAS Grid Migration Prasoon Sangwan, Piyush Singh, Tanuj Gupta TATA Consultancy Services Ltd. ABSTRACT When SAS users move from traditional departmental
More informationOracle Login Max Length Table Name 11g Column Varchar2
Oracle Login Max Length Table Name 11g Column Varchar2 Get max(length(column)) for all columns in an Oracle table tables you are looking at BEGIN -- loop through column names in all_tab_columns for a given
More informationProgramming Beyond the Basics. Find() the power of Hash - How, Why and When to use the SAS Hash Object John Blackwell
Find() the power of Hash - How, Why and When to use the SAS Hash Object John Blackwell ABSTRACT The SAS hash object has come of age in SAS 9.2, giving the SAS programmer the ability to quickly do things
More informationSAS 9.3 LIBNAME Engine for DataFlux Federation Server
SAS 9.3 LIBNAME Engine for DataFlux Federation Server User s Guide SAS Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2012. SAS 9.3 LIBNAME Engine for
More informationPharmaceuticals, Health Care, and Life Sciences
Successful Lab Result Conversion for LAB Analysis Data with Minimum Effort Pushpa Saranadasa, Merck & Co., Inc. INTRODUCTION In the pharmaceutical industry, the statistical results of a clinical trial's
More informationUsing SAS Files. Introduction CHAPTER 5
123 CHAPTER 5 Using SAS Files Introduction 123 SAS Data Libraries 124 Accessing SAS Files 124 Advantages of Using Librefs Rather than OpenVMS Logical Names 124 Assigning Librefs 124 Using the LIBNAME Statement
More informationStatistics, Data Analysis & Econometrics
ST009 PROC MI as the Basis for a Macro for the Study of Patterns of Missing Data Carl E. Pierchala, National Highway Traffic Safety Administration, Washington ABSTRACT The study of missing data patterns
More informationSAS/Warehouse Administrator Usage and Enhancements Terry Lewis, SAS Institute Inc., Cary, NC
SAS/Warehouse Administrator Usage and Enhancements Terry Lewis, SAS Institute Inc., Cary, NC ABSTRACT SAS/Warehouse Administrator software makes it easier to build, maintain, and access data warehouses
More informationTroy Martin Hughes ABSTRACT INTRODUCTION
MWSUG 2016 - Paper 38 A Failure To EXIST: Why Testing for Data Set Existence with the EXIST Function Alone Is Inadequate for Serious Software Development in Asynchronous, Multiuser, and Parallel Processing
More informationParser Tools: lex and yacc-style Parsing
Parser Tools: lex and yacc-style Parsing Version 6.11.0.6 Scott Owens January 6, 2018 This documentation assumes familiarity with lex and yacc style lexer and parser generators. 1 Contents 1 Lexers 3 1.1
More informationPerl Regular Expression The Power to Know the PERL in Your Data
ABSTRACT MWSUG 2018 Paper SB-145 Perl Regular Expression The Power to Know the PERL in Your Data Kaushal Chaudhary, Eli Lilly and Company, Indianapolis, IN Dhruba Ghimire, Eli Lilly and Company, Indianapolis,
More informationTop 10 SAS Functions in A brief summary of SAS Communities Survey - by Flora Fang Liu
Top 10 SAS Functions in 2017 A brief summary of SAS Communities Survey - by Flora Fang Liu 1 What are SAS Functions? Why use SAS Functions? What? SAS functions perform computations, data manipulation,
More informationFull file at
SQL for SQL Server 1 True/False Questions Chapter 2 Creating Tables and Indexes 1. In order to create a table, three pieces of information must be determined: (1) the table name, (2) the column names,
More informationMoving and Accessing SAS. 9.1 Files
Moving and Accessing SAS 9.1 Files The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2004. Moving and Accessing SAS 9.1 Files. Cary, NC: SAS Institute Inc. Moving and
More informationFifteen Functions to Supercharge Your SAS Code
MWSUG 2017 - Paper BB071 Fifteen Functions to Supercharge Your SAS Code Joshua M. Horstman, Nested Loop Consulting, Indianapolis, IN ABSTRACT The number of functions included in SAS software has exploded
More informationBase and Advance SAS
Base and Advance SAS BASE SAS INTRODUCTION An Overview of the SAS System SAS Tasks Output produced by the SAS System SAS Tools (SAS Program - Data step and Proc step) A sample SAS program Exploring SAS
More informationNow That You Have Your Data in Hadoop, How Are You Staging Your Analytical Base Tables?
Paper SAS 1866-2015 Now That You Have Your Data in Hadoop, How Are You Staging Your Analytical Base Tables? Steven Sober, SAS Institute Inc. ABSTRACT Well, Hadoop community, now that you have your data
More informationImproving Your Relationship with SAS Enterprise Guide Jennifer Bjurstrom, SAS Institute Inc.
ABSTRACT Paper BI06-2013 Improving Your Relationship with SAS Enterprise Guide Jennifer Bjurstrom, SAS Institute Inc. SAS Enterprise Guide has proven to be a very beneficial tool for both novice and experienced
More informationSAS File Management. Improving Performance CHAPTER 37
519 CHAPTER 37 SAS File Management Improving Performance 519 Moving SAS Files Between Operating Environments 520 Converting SAS Files 520 Repairing Damaged Files 520 Recovering SAS Data Files 521 Recovering
More informationUSING SAS HASH OBJECTS TO CUT DOWN PROCESSING TIME Girish Narayandas, Optum, Eden Prairie, MN
Paper RF-12-2014 USING SAS HASH OBJECTS TO CUT DOWN PROCESSING TIME Girish Narayandas, Optum, Eden Prairie, MN ABSTRACT Hash tables are in existence since SAS 9 version and are part of data step programming.
More informationSo, Your Data are in Excel! Ed Heaton, Westat
Paper AD02_05 So, Your Data are in Excel! Ed Heaton, Westat Abstract You say your customer sent you the data in an Excel workbook. Well then, I guess you'll have to work with it. This paper will discuss
More informationUncommon Techniques for Common Variables
Paper 11863-2016 Uncommon Techniques for Common Variables Christopher J. Bost, MDRC, New York, NY ABSTRACT If a variable occurs in more than one data set being merged, the last value (from the variable
More informationJUST PASSING THROUGH OR ARE YOU? DETERMINE WHEN SQL PASS THROUGH OCCURS TO OPTIMIZE YOUR QUERIES Misty Johnson Wisconsin Department of Health
JUST PASSING THROUGH OR ARE YOU? DETERMINE WHEN SQL PASS THROUGH OCCURS TO OPTIMIZE YOUR QUERIES Misty Johnson Wisconsin Department of Health Services, Madison, WI Outline SAS/ACCESS SQL Pass Through Facility
More informationSo Much Data, So Little Time: Splitting Datasets For More Efficient Run Times and Meeting FDA Submission Guidelines
Paper TT13 So Much Data, So Little Time: Splitting Datasets For More Efficient Run Times and Meeting FDA Submission Guidelines Anthony Harris, PPD, Wilmington, NC Robby Diseker, PPD, Wilmington, NC ABSTRACT
More informationABSTRACT DATA CLARIFCIATION FORM TRACKING ORACLE TABLE INTRODUCTION REVIEW QUALITY CHECKS
Efficient SAS Quality Checks: Unique Error Identification And Enhanced Data Management Analysis Jim Grudzinski, Biostatistics Manager Of SAS Programming Covance Periapproval Services Inc, Radnor, PA ABSTRACT
More informationInterleaving a Dataset with Itself: How and Why
cc002 Interleaving a Dataset with Itself: How and Why Howard Schreier, U.S. Dept. of Commerce, Washington DC ABSTRACT When two or more SAS datasets are combined by means of a SET statement and an accompanying
More informationA Format to Make the _TYPE_ Field of PROC MEANS Easier to Interpret Matt Pettis, Thomson West, Eagan, MN
Paper 045-29 A Format to Make the _TYPE_ Field of PROC MEANS Easier to Interpret Matt Pettis, Thomson West, Eagan, MN ABSTRACT: PROC MEANS analyzes datasets according to the variables listed in its Class
More informationSAS Data Explorer 2.1: User s Guide
SAS Data Explorer 2.1: User s Guide Working with SAS Data Explorer Understanding SAS Data Explorer SAS Data Explorer and the Choose Data Window SAS Data Explorer enables you to copy data to memory on SAS
More informationWKn Chapter. Note to UNIX and OS/390 Users. Import/Export Facility CHAPTER 9
117 CHAPTER 9 WKn Chapter Note to UNIX and OS/390 Users 117 Import/Export Facility 117 Understanding WKn Essentials 118 WKn Files 118 WKn File Naming Conventions 120 WKn Data Types 120 How the SAS System
More informationMulti-Threaded Reads in SAS/Access for Relational Databases Sarah Whittier, ISO New England, Holyoke, MA
Multi-Threaded Reads in SAS/Access for Relational Databases Sarah Whittier, ISO New England, Holyoke, MA ABSTRACT Multi-threading was implemented in SAS 9. This new feature affects the performance of certain
More informationOUT= IS IN: VISUALIZING PROC COMPARE RESULTS IN A DATASET
OUT= IS IN: VISUALIZING PROC COMPARE RESULTS IN A DATASET Prasad Ilapogu, Ephicacy Consulting Group; Masaki Mihaila, Pfizer; ABSTRACT Proc compare is widely used in the pharmaceutical world to validate
More informationAPPENDIX 4 Migrating from QMF to SAS/ ASSIST Software. Each of these steps can be executed independently.
255 APPENDIX 4 Migrating from QMF to SAS/ ASSIST Software Introduction 255 Generating a QMF Export Procedure 255 Exporting Queries from QMF 257 Importing QMF Queries into Query and Reporting 257 Alternate
More informationCV2ODBC Procedure. Overview. CV2ODBC Procedure Syntax APPENDIX 4
263 APPENDIX 4 CV2ODBC Procedure Overview 263 CV2ODBC Procedure Syntax 263 PROC CV2ODBC 264 FROM VIEW Statement 264 TO VIEW Statement 264 DSN Statement 265 UID Statement 265 PASSWORD Statement 265 SAVE
More informationA Macro to Keep Titles and Footnotes in One Place
CC25 ABSTRACT A Macro to Keep Titles and Footnotes in One Place John Morrill, Quintiles, Inc., Kansas City, MO A large project with titles and footnotes in each separate program can be cumbersome to maintain.
More informationRelease Notes.
ReleaseNotesTitle.fm) Release Notes InterBase 2009 www.embarcadero.com 2008 Embarcadero Technologies, Inc. Embarcadero, the Embarcadero Technologies logos, and all other Embarcadero Technologies product
More informationThe new SAS 9.2 FCMP Procedure, what functions are in your future? John H. Adams, Boehringer Ingelheim Pharmaceutical, Inc.
PharmaSUG2010 - Paper AD02 The new SAS 9.2 FCMP Procedure, what functions are in your future? John H. Adams, Boehringer Ingelheim Pharmaceutical, Inc., Ridgefield, CT ABSTRACT Our company recently decided
More informationEfficiently Join a SAS Data Set with External Database Tables
ABSTRACT Paper 2466-2018 Efficiently Join a SAS Data Set with External Database Tables Dadong Li, Michael Cantor, New York University Medical Center Joining a SAS data set with an external database is
More informationA SAS/AF Application for Parallel Extraction, Transformation, and Scoring of a Very Large Database
Paper 11 A SAS/AF Application for Parallel Extraction, Transformation, and Scoring of a Very Large Database Daniel W. Kohn, Ph.D., Torrent Systems Inc., Cambridge, MA David L. Kuhn, Ph.D., Innovative Idea
More informationChapter 1 INTRODUCTION. SYS-ED/ Computer Education Techniques, Inc.
Chapter 1 INTRODUCTION SYS-ED/ Computer Education Techniques, Inc. Objectives You will learn: Facilities and features of PL/1. Structure of programs written in PL/1. Data types. Storage classes, control,
More informationPlease don't Merge without By!!
ABSTRACT Please don't Merge without By!! Monal Kohli Have you ever merged datasets and forgotten a by Statement, looked at the results and thought wow -- 100% match but when you started validating the
More informationMoving and Accessing SAS 9.2 Files
Moving and Accessing SAS 9.2 Files The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2008. Moving and Accessing SAS 9.2 Files. Cary, NC: SAS Institute Inc. Moving and
More informationDeveloping SAS Studio Repositories
Paper SAS6140-2016 Developing SAS Studio Repositories Marie Dexter, Swapnil Ghan, Michael Monaco, and Amy Peters, SAS Institute Inc., Cary, NC ABSTRACT This paper covers developing SAS Studio repositories.
More informationPhUse Practical Uses of the DOW Loop in Pharmaceutical Programming Richard Read Allen, Peak Statistical Services, Evergreen, CO, USA
PhUse 2009 Paper Tu01 Practical Uses of the DOW Loop in Pharmaceutical Programming Richard Read Allen, Peak Statistical Services, Evergreen, CO, USA ABSTRACT The DOW-Loop was originally developed by Don
More informationPharmaSUG China Mina Chen, Roche (China) Holding Ltd.
PharmaSUG China 2017-50 Writing Efficient Queries in SAS Using PROC SQL with Teradata Mina Chen, Roche (China) Holding Ltd. ABSTRACT The emergence of big data, as well as advancements in data science approaches
More informationEXAMPLE 3: MATCHING DATA FROM RESPONDENTS AT 2 OR MORE WAVES (LONG FORMAT)
EXAMPLE 3: MATCHING DATA FROM RESPONDENTS AT 2 OR MORE WAVES (LONG FORMAT) DESCRIPTION: This example shows how to combine the data on respondents from the first two waves of Understanding Society into
More informationBest Practice for Creation and Maintenance of a SAS Infrastructure
Paper 2501-2015 Best Practice for Creation and Maintenance of a SAS Infrastructure Paul Thomas, ASUP Ltd. ABSTRACT The advantage of using metadata to control and maintain data and access to data on databases,
More informationIndenting with Style
ABSTRACT Indenting with Style Bill Coar, Axio Research, Seattle, WA Within the pharmaceutical industry, many SAS programmers rely heavily on Proc Report. While it is used extensively for summary tables
More informationA Side of Hash for You To Dig Into
A Side of Hash for You To Dig Into Shan Ali Rasul, Indigo Books & Music Inc, Toronto, Ontario, Canada. ABSTRACT Within the realm of Customer Relationship Management (CRM) there is always a need for segmenting
More informationBASICS BEFORE STARTING SAS DATAWAREHOSING Concepts What is ETL ETL Concepts What is OLAP SAS. What is SAS History of SAS Modules available SAS
SAS COURSE CONTENT Course Duration - 40hrs BASICS BEFORE STARTING SAS DATAWAREHOSING Concepts What is ETL ETL Concepts What is OLAP SAS What is SAS History of SAS Modules available SAS GETTING STARTED
More informationPerformance Considerations
149 CHAPTER 6 Performance Considerations Hardware Considerations 149 Windows Features that Optimize Performance 150 Under Windows NT 150 Under Windows NT Server Enterprise Edition 4.0 151 Processing SAS
More informationPaper William E Benjamin Jr, Owl Computer Consultancy, LLC
Paper 025-2009 So, You ve Got Data Enterprise Wide (SAS, ACCESS, EXCEL, MySQL, and Others); Well, Let SAS Enterprise Guide Software Point-n-Click Your Way to Using It William E Benjamin Jr, Owl Computer
More informationGetting it Done with PROC TABULATE
ABSTRACT Getting it Done with PROC TABULATE Michael J. Williams, ICON Clinical Research, San Francisco, CA The task of displaying statistical summaries of different types of variables in a single table
More informationLet Hash SUMINC Count For You Joseph Hinson, Accenture Life Sciences, Berwyn, PA, USA
ABSTRACT PharmaSUG 2014 - Paper CC02 Let Hash SUMINC Count For You Joseph Hinson, Accenture Life Sciences, Berwyn, PA, USA Counting of events is inevitable in clinical programming and is easily accomplished
More informationIntroduction to Computer Science and Business
Introduction to Computer Science and Business The Database Programming with PL/SQL course introduces students to the procedural language used to extend SQL in a programatic manner. This course outline
More informationSAS Viya 3.2: Self-Service Import
SAS Viya 3.2: Self-Service Import About Self-Service Import Overview of Self-Service Import Self-service import offers an easy way to bring data into the SAS Cloud Analytic Services (CAS) environment.
More informationSAS Studio: A New Way to Program in SAS
SAS Studio: A New Way to Program in SAS Lora D Delwiche, Winters, CA Susan J Slaughter, Avocet Solutions, Davis, CA ABSTRACT SAS Studio is an important new interface for SAS, designed for both traditional
More information