Seeking, Mapping, and Fuzzy Merging Data Structures in a Networked Environment Haftan Eckholdt, Albert Einstein College of Medicine, Bronx, NY

Transcription

1 Seeking, Mapping, and Fuzzy Merging Data Structures in a Networked Environment Haftan Eckholdt, Albert Einstein College of Medicine, Bronx, NY ABSTRACT The challenges of long term, multi-site projects include that acquisition, management, and analysis of data (files) distributed across many workstations from many points in time. Finding, reading and reconciling these files often involves data loss rates beyond tolerance for tracking and analysis in longitudinal / repeated measurement projects. SAS MACRO and the SAS X command can be used like a robot to search drives and directories across networks in order to seek and map file structures. The product of file mapping can then be used to read eligible files and fuzzy merge them using various spatio-temporal criteria to produce final (wide or deep) data structures linked with highest-to-lowest probability of match. Several examples of these processes will be demonstrated including: (1) the first part of this process to document hard drives across a small Novell / NT network, (2) the second part of this process to merge very large lists in a random digit dial computer aided telephone interview (RDD-CATI) laboratory, and (3) a combination of all of the above to openly seek, map, acquire, merge and analyze data in a laboratory of functional neuroanatomy/neurophysiology. INTRODUCTION Researchers today are challenged with managing rapidly growing multisited data structures, a trend that is likely to accelerate with internet and intranet evolution and usage. It is not unusual for analysts to manage and manipulate hundreds of thousands of files of varying formats residing in many known and unknown places. and review hundreds of pages of images, wave forms, topographic densities, or metabolic events, each representing one spatial perspective at one point in time in one condition from one unit of observation. A typical experiment or project often involves thousands of files, each with thousands of data points. It is not unusual for laboratories to collect a gigabyte of relevant data points on each subject or event. Nor is it unusual for the drive, directory (sub-sub-sub), and file naming schemes to have no symmetric logic, and no documentation. Some researchers have no idea where data have been stored and what they are named. This reality is not a sign of bad science, just bad technical management. And bad technical management means that files cannot be found for analysis. Lost data begins to look like bad science as many great hypotheses fall between the cracks of time and space along with their data. To date, there are no self-organizing software systems that can take the place of good technical management. So scientists are left to their own (de)vices. PROPOSED SOLUTIONS This manuscript describes the four steps of seeking data files, mapping data files, acquiring data, and merging data. These steps can be used in many applications but the examples will come from efforts to find, acquire, and merge data from the Einstein Aging Study, a longitudinal study of Alzhemier s Disease funded by the National Institute on Aging. SEEKING DATA FILES The first step, and perhaps only step, in seeking data files is to identify the hard drive(s) to be mapped. This is a relatively simple boot strap to the entire process, but it may require that the user map networked drives to the local workstation. The reader should note here that the user on the SAS processing workstation must have the privileges of a network administrator to do much of this work, and that the hard drives to be mapped have already be assigned share status, along with users and passwords. These procedures will not map non-local drives invasively without prior permissions and explicit privileges being established. The code below starts by turning off the X command feature with opens an Operating System window and waits for the user to exit the window before continuing to the next step. This is a critical start since hundreds of thousands of commands may be delivered to the local Operating System. Waiting for user input would turn an 8 hour job into an 8 month job. The following code then creates two SAS libraries (a temp library, and a long term library). This code then turns off the SAS log which will crash almost any operating system hard disk array with the files mapped and merged goes into the thousands. This code then cycles through a SAS MACRO procedure that names drive N to be mapped. OPTIONS PAGENO=1 NOXWAIT LIBNAME WORK C:\TEMP\SAS\ LIBNAME NESUG C:\NESUG\2000\DATA\ FILENAME NOLOG DUMMY PROC PRINTTO LOG=NOLOG %LET DRIVE = N The next step uses the SAS X command which sends the quoted string to the operating system. This code simulates opening a local operating system window and typing DIR N:\ > C:\NESUG\2000\DATA\DIR1.TXT This text uses the Microsoft DOS dir command to list the contents of the N: drive root directory. The most important feature of this command is the > sign which tell the operating system to dump the output of the dir command to the following file rather than the screen. In this case, the target file is call DIR1.txt, and it will consist of ASCII text and reside in the NESUG library on the local C drive. X "DIR L:\EAS\DATA\ARCHIVE > D:\AECOM\EAS\DATA\UNARCH\DIR1.TXT" The output file looks something like the following: Volume in drive N has no label. Volume Serial Number is 164A-18D6 Directory of N:\ 02/26/00 08:15a <DIR> FOUND /27/00 05:06p <DIR> I386 01/27/00 05:10p <DIR> SP3 01/27/00 05:23p <DIR> WINNT 01/27/00 05:21p <DIR> LRVR 06/28/00 01:46p 212,860,928 pagefile.sys 01/27/00 05:30p <DIR> Program Files 01/27/00 05:31p <DIR> TEMP 01/27/00 05:33p <DIR> WinUtils 01/27/00 05:34p <DIR> ToshUtil 01/27/00 05:34p <DIR> hddpwdnt 01/28/00 02:43p <DIR> Win32App 01/28/00 02:48p <DIR> SAS 01/28/00 03:21p <DIR> Multimedia Files 01/28/00 03:31p <DIR> lotus 01/28/00 04:18p <DIR> PASS60 02/18/00 07:45p <DIR> Iomg_NT 06/30/00 12:48p 0 DIR1.TXT 18 File(s) 212,860,928 bytes 88,768,512 bytes free MAPPING DATA FILES As one can see, the N drive is someone s C: drive since it contains all of the Operating System and software

2 directories that one would expect to find on the root. The following code will perform file mapping by reading the DIR output file as a data file. In this example, the SAS binary file will be called DIR1 and stored in the NESUG library. In Windows NT 4.0 workstation, the variables read in include file creation date, file creation time, including am/pm, the directory descriptor whether or not the object is a file or directory, and finally, the file or directory name. None of these data appear until the 8 th line, so the FIRSTOBS option must be used to skip over the file header garbage. DATA NESUG.DIR1 INFILE "D:\AECOM\EAS\DATA\UNARCH\DIR1.TXT" FIRSTOBS = 8 FDATE MMDDYY8. FTIME TIME7. AMPM $ FDIR $ FNAME $ IF FDIR NE "<DIR>" THEN DELETE PROC SORT BY FDIR DATA _NULL_ CALL SYMPUT ("SAMPLE",COUNT) STOP SET EAS.DIR1 NOBS=COUNT run Prior to using this code, which was written and optimized to run on a Microsoft Windows NT 4.0 workstation (SP3), the user must submit the quoted X command string and verify the structure of the DIR output file. The next lines in the code the perform the special task of acquiring the directories only, and creating a temporary SAS data file that only contains the directory tagged observations from the DIR output file. The number of directories in the DIR output file is also saved as a variable called sample to be used in the next set of MACRO procedures call Look. This macro loops around once for each observation (directory) in the DIR output file (ASCII version called DIR1.TXT and SAS version call DIR1.SD2) using the %DO loop until the %END at the bottom. For each directory on N:, a variable is created call SDIR which is space trimmed version of FNAME. This will allow the MACRO LOOK to then perform the same DIR X command on each and every directory in DIR1.SD2 which will be called DIR2.TXT and DIR2.SD2 respectively. %MACRO LOOK %DO L = 1 %TO &SAMPLE DATA EAS.DIR1 SET EAS.DIR1 IF _N_ = &L THEN CALL SYMPUT ("SDIR",TRIM(FNAME)) RUN X "DIR N:\&SDIR > C:\NESUG\2000\DATA\DIR2.TXT" DATA EAS.DIR2 INFILE "C:\NESUG\2000\DATA\DIR2.TXT" FIRSTOBS = 8 FDATE2 MMDDYY8. FTIME2 TIME7. AMPM2 $ FDIR2 $ FNAME2 $ DIR1 = "&SDIR" IF FDIR2 NE "<DIR>" THEN DELETE PROC SORT BY FNAME2 DATA EAS.DIR2 SET EAS.DIR2 BY FNAME2 CALL SYMPUT ("SDIR2",PUT(FNAME2,Z8.)) RUN At this point, the reader should understand that this process can continue infinitely until there are not more subdirectories, with minor edits to the macro syntax and logic. A few additional procedures can then be used to bring about a single file structure that contains all of the files, and all of their directory tree information. Note that the first time around, a file must be created rather than appended (%* commented out below). PROC TRANSPOSE DATA=EAS.DIR2 OUT=EAS.DIR2H NAME=File PREFIX=FILE VAR FNAME %*DATA NESUG.ARCHIVE data NESUG.dir2h ATTRIB DIR1 LENGTH=$9 attrib DIR2 LENGTH=$8 SET NESUG.DIR2H DIR1 = "&DIR1" DIR2 = "&DIR2" PROC APPEND BASE=NESUG.ARCHIVE DATA=NESUG.DIR2H FORCE %END %MEND LOOK %LOOK In this last step, a frequency distribution is established about the number of directories and files. DATA NESUG.ARCHIVE SET NESUG.ARCHIVE NFILE = 0 IF FILE1 NE " " THEN NFILE = NFILE +1 IF FILE2 NE " " THEN NFILE = NFILE +1 IF FILE3 NE " " THEN NFILE = NFILE +1 IF FILE4 NE " " THEN NFILE = NFILE +1 IF FILE5 NE " " THEN NFILE = NFILE +1 IF FILE6 NE " " THEN NFILE = NFILE +1 IF FILE7 NE " " THEN NFILE = NFILE +1 IF FILE8 NE " " THEN NFILE = NFILE +1 IF FILE9 NE " " THEN NFILE = NFILE +1 IF FILE10 NE " " THEN NFILE = NFILE +1 PROC MEANS N MIN MAX MEAN SUM TITLE2 "ARCHIVE READING FOR NESUG 2000" VAR NFILE PROC FREQ TABLES NFILE RUN A next step in file mapping might also involve a summary of

3 all file names and file types. This will help in deciding which kinds of files will be acquired next. %macro LOOK2 %DO A = 1 %TO 10 DATA FILE&A SET NESUG.ARCHIVE FILE = FILE&A DOT = INDEX(FILE,".") FILEN = SUBSTR(FILE,1,DOT-1) IF DOT > 0 THEN FILET = SUBSTR(FILE,DOT+1) KEEP FILE DIR1 DIR2 FILE FILEN FILET %END DATA NESUG.ARCHFILE SET %DO B = 1 %TO 10 FILE&B %END %MEND LOOK2 %LOOK2 PROC SORT BY DIR1 DIR2 FILET PROC FREQ TABLES DIR1 DIR2 PROC FREQ TABLES FILET BY DIR1 RUN ACQUIRING DATA It is now necessary to clear the SAS temp libraries to prevent any rereading of data structures. A selected group of directories and file types will now be read. Accordingly, the reader must know the format and syntax of the files to be read. In this example, a series of vertical data files will be read in from the SAS directory. These files will then be read into a single vertical SAS data file for analysis. PROC DATASETS LIBRARY=WORK KILL DATA NESUG.FLA SET NESUG.ARCHFILE IF DIR1 = "SAS" DIR2 = WORK FILET = "FL" PROC SORT NODUPKEY BY FILEN FILET RUN DATA _NULL_ CALL SYMPUT ("SAMPLE",COUNT) STOP SET NESUG.FLA NOBS=COUNT run %MACRO LOOK3 %DO F = 1 %TO &SAMPLE DATA NESUG.FLA SET NESUG.FLA IF _N_ = &F THEN CALL SYMPUT ("DIR1",TRIM(DIR1)) IF _N_ = &F THEN CALL SYMPUT ("DIR2",TRIM(DIR2)) IF _N_ = &F THEN CALL SYMPUT ("FILE",TRIM(FILEN)) DATA FLA&F INFILE "N:\&DIR1\&DIR2\LEARN\&FILE..FL" DLM=',' DSD recfm=v LRECL=100 Last : $CHAR15. First : $CHAR15. / ID / Age / YrWave : $CHAR3. / FCSR : $CHAR12. / Date : $CHAR8. / FL0701 / FL0801 / CT01 $ TR01 $ FL0103 FL0104 / RC0101 XT0101 : $CHAR12. XT0102 : $CHAR12. XT0103 : $CHAR12. RT0104 / CT02 $ TR02 $ FL0203 FL0204 / ETC ARCHID = &DIR1 ARCHWA = &DIR2 ARCHFI = "&FILE" %END DATA EAS.FLA SET %DO F = 1 %TO &SAMPLE FLA&F %END %MEND LOOK3 %LOOK3 Further along in this process, some variables need to be combined for modeling. %MACRO LOOK4 %LET VAR = %LET DIG = %DO V = 1 %TO 16 %DO D = 1 %TO 3 S%SCAN(&VAR,&V)%SCAN(&DIG,&D) = INDEX(XT%SCAN(&VAR,&V)%SCAN(&DIG,&D)," ") RT%SCAN(&VAR,&V)%SCAN(&DIG,&D) = SUBSTR(XT%SCAN(&VAR,&V)%SCAN(&DIG,&D),1, S%SCAN(&VAR,&V)%SCAN(&DIG,&D)-1) RC%SCAN(&VAR,&V)%SCAN(&DIG,&D) = SUBSTR(XT%SCAN(&VAR,&V)%SCAN(&DIG,&D), S%SCAN(&VAR,&V)%SCAN(&DIG,&D)+1) %END %END %MEND LOOK4 %LOOK4 YEAR = SUBSTR(YRWAVE,1,2) WAVE = SUBSTR(YRWAVE,3,1)

4 MM = SUBSTR(DATE,1,2) DD = SUBSTR(DATE,4,2) YY = SUBSTR(DATE,7,2) EVENT = 1 IF ID = LAG1(ID) YRWAVE=LAG1(YRWAVE) THEN EVENT = 2 PROC FORMAT VALUE EV 1 = "Instant" 2 = "Delay" PROC PRINT TITLE3 "FL A SET Data Acquisition" PROC FREQ TABLES EVENT FORMAT EVENT EV. RUN MERGING DATA Finally, we reach the process of fuzzy merging. In this example, fuzzy merging comes from a different part of the study the development of the CATI RDD phone sample. In this example, the source file contains the name and address of a potential subject, while a separate file contains the name, address, and phone number of all people in the white pages directory for the same county. The essential logic of the system involves the separation of source names into character string groups based on the length of the source. That way, a short last name of 4 characters is only merged with potential targets that are of matching length. This initial process greatly decreases total CPU time (this code ran in serial process for approximately 60 days) and increases accuracy (a number was found for about 50% of the sources an accuracy in excess of 99%). LIBNAME EAS LIBNAME TEMP D:\AECOM\EAS\HCFA98\DATA\ F:\TEMP\ FILENAME NOLOG DUMMY PROC PRINTTO LOG=NOLOG PRINT= F:\TEMP\sample.LST ** READ IN PHONE BOOK ASCII DATA FILE DATA EAS.PROCD2B INFILE D:\AECOM\EAS\HCFA98\DATA\PROCD2B.PRN LRECL=74 PAD NAMEP $ 1-20 ADDPH $ ZIPP PHONE $ * SET MARKER PROCD = 1 * GRAB FIRST WORD AS LAST NAME SECOND WORD FIRST NAME LAST=SCAN(NAMEP,1) FIRPH=SCAN(NAMEP,2) SECPH=SCAN(NAMEP,3) * FIND LENGTH OF EACH NAME CUTLPH=LENGTH(LAST) CUTFPH=LENGTH(FIRPH) LABEL ADDPH = Phone Address NAMEP = Phone Name ** READ IN HCFA ASCII DATA FILE DATA EAS.HCFA98 INFILE D:\AECOM\EAS\HCFA98\DATA\FOREIGN.TXT LRECL=205 PAD NAMEH $ ADDHC $ BDYY BDDDD DDYY DDDDD ZIPH SEX 187 RACE 188 * SET MARKER HCFA=1 * GRAB FIRST WORD AS LAST NAME SECOND WORD FIRST NAME LAST=SCAN(NAMEH,1) FIRHC=SCAN(NAMEH,2) SECHC=SCAN(NAMEH,3) * FIND LENGTH OF EACH NAME CUTLHC=LENGTH(LAST) CUTFHC=LENGTH(FIRHC) AGE = 98-BDYY + ((365 - BDDDD)/365) IF AGE < 55 THEN AGE = AGE AGE80_01=1 IF AGE < 80 THEN AGE80_01 = 0 DEATH = 0 IF DDYY >. THEN DEATH = 1 RACE2 = RACE IF RACE2 = 0 THEN RACE2 = 3 IF RACE2 > 2 THEN RACE2 = 3 PROC FORMAT VALUE DEA 0= Alive 1= Dead VALUE AGE 0= <80 1= 80+ VALUE SEX 1= Male 2= Female VALUE RAC 1= White 2= Black 3= Other VALUE MAT 0= No Match 1= Match LABEL AGE80_01 = Age Strata ADDHC = HCFA Address NAMEH = HCFA Name RACE2 = Race PROC CONTENTS TITLE3 "HCFA LIST DESCRIPTION" PROC FREQ TABLES DEATH TABLES RACE2 TABLES AGE80_01 TABLES SEX FORMAT DEATH DEA. RACE2 RAC. AGE80_01 AGE. SEX SEX. PROC UNIVARIATE PLOT NORMAL VAR AGE *** BEGIN CUTTING PHONE DATA * NOTE THERE ARE LAST NAMES WITH 1 TO 17 CHARACTERS %MACRO PHONCUT

5 %DO P = 1 %TO 17 %* MAKE DATA SET FOR EACH LAST NAME LENGTH / FIRST NAME DATA EAS.PHONB&P SET EAS.PROCD2B %END %MEND PHONCUT IF CUTLPH = &P ** BEGIN CUTTING HCFA DATA * NOTE THERE ARE LAST NAMES WITH 1 TO 19 CHARACTERS %MACRO HCFACUT %DO H = 1 %TO 19 %* MAKE DATA SET FOR EACH LAST NAME LENGTH / FIRST NAME DATA EAS.HCFAB&H SET EAS.HCFA98 ** RUN MACROS %PHONCUT %HCFACUT %END %MEND HCFACUT IF CUTLHC = &H Note here that at this point in the process, the data structures (source and target candidates) are completed and the process of repeated merging begins. In order to maximize successful merging, each source observation must be merged with all same length targets. This is very CPU intensive (for more than 100,000 sources)! Note too that this process is initially conducted on the very first source candidate to establish the files, the looped through the remaining source candidates using the APPEND statement. ***** BEGIN BANGING THE FULL NAME DATA STRUCTURES * CLEAR THE WORKING BASE APPENDING DATASET data temp.hcphbase set %MACRO SAMPLE %DO B = 1 %TO 17 data _null_ set eas.hcfab&b call symput( last,trim(left(_n_))) %* CALL PHONE HCFA INTO BEING DATA PH SET EAS.PHONB&B PROC SORT BY LAST DATA HC SET EAS.HCFAB&B %* GRAB FIRST HCFA OBS IF _N_ = 1 stack=1 %* MERGE FIRST HCFA OBS WITH PHONE DATA SET MATCH = 1 DATA HCPH MERGE HC PH BY LAST DATA temp.hcphb&b SET HCPH MATCH = 0 IF HCFA = 1 PROCD = 1 THEN IF MATCH=1 OR MATCH=0 HCFA = 1 This part of the code develops a measure of match accuracy based on the number of characters in the name that match in the source and the target, as well as the number of characters in the address that match in the source and the target. Each source observation then has a dataset of target matched candidates that is sorted for saving the highest match. %* RUN THE MATCH COUNTING [DIS_TANCE] SEQUENCE DISNAM = 0 substr(firph,1,1) THEN disnam = disnam + 1 substr(firph,2,1) THEN disnam = disnam + 1 substr(firph,2,1) substr(firph,3,1) THEN disnam = disnam + 1 substr(firph,2,1) substr(firph,3,1) substr(firhc,4,1) = substr(firph,4,1) THEN disnam = disnam + 1 substr(firph,2,1) substr(firph,3,1) substr(firhc,4,1) = substr(firph,4,1) substr(firhc,5,1) = substr(firph,5,1) THEN disnam = disnam + 1 DISNAM2 = 0 substr(secph,1,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) substr(sechc,4,1) = substr(secph,4,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) substr(sechc,4,1) = substr(secph,4,1)

6 substr(sechc,5,1) = substr(secph,5,1) THEN disnam2= disnam2+ 1 DISADD = 0 THEN disadd = disadd + 1 THEN disadd = disadd + 1 substr(addhc,3,1) = substr(addph,3,1) THEN disadd = disadd + 1 substr(addhc,3,1) = substr(addph,3,1) substr(addhc,4,1) = substr(addph,4,1) THEN disadd = disadd + 1 substr(addhc,3,1) = substr(addph,3,1) substr(addhc,4,1) = substr(addph,4,1) substr(addhc,5,1) = substr(addph,5,1) THEN disadd = disadd + 1 proc sort by last disadd disnam disnam2 data temp.hcphb&b set temp.hcphb&b by last disadd disnam disnam2 if last.last %* SLOPE NOW INTO THE REMAINING OBS OF THE HCFA DATASET %* BASE DATASET IS HCFA THE ADDED DATA COMES FROM PROCD %* CREATE COMMON MERGE NAME / ADDRESS VARS IN EACH DATASET OF LOW TO HIGH MATCH = 1 and %MACRO BANGER %DO C = 2 %TO &LAST DATA HC SET EAS.HCFAB&B IF _N_ = &C stack=&c DATA HCPH MERGE HC PH BY LAST DATA temp.hcphbas SET HCPH IF MATCH = 0 IF HCFA = 1 PROCD = 1 THEN MATCH=1 or MATCH=0 HCFA=1 DISNAM = 0 substr(firph,1,1) THEN disnam = disnam + 1 substr(firph,1,1) substr(firph,2,1) THEN disnam = disnam + 1 substr(firph,1,1) and substr(firph,2,1) substr(firph,3,1) THEN disnam = disnam + 1 substr(firph,2,1) substr(firph,3,1) substr(firhc,4,1) = substr(firph,4,1) THEN disnam = disnam + 1 substr(firph,2,1) substr(firph,3,1) substr(firhc,4,1) = substr(firph,4,1) substr(firhc,5,1) = substr(firph,5,1) THEN disnam = disnam + 1 DISNAM2 = 0 substr(secph,1,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) substr(sechc,4,1) = substr(secph,4,1) THEN disnam2= disnam2+ 1 substr(secph,2,1) substr(secph,3,1) substr(sechc,4,1) = substr(secph,4,1) substr(sechc,5,1) = substr(secph,5,1) THEN disnam2= disnam2+ 1 DISADD = 0 IF substr(addhc,1,1) = substr(addph,1,1) THEN disadd = disadd + 1 IF substr(addhc,1,1) = substr(addph,1,1) substr(addhc,2,1) = substr(addph,2,1) THEN disadd = disadd + 1 IF substr(addhc,1,1) = substr(addph,1,1) substr(addhc,2,1) = substr(addph,2,1) substr(addhc,3,1) = substr(addph,3,1) THEN disadd = disadd + 1 IF substr(addhc,1,1) = substr(addph,1,1) substr(addhc,2,1) = substr(addph,2,1) substr(addhc,3,1) =

7 substr(addph,3,1) substr(addhc,4,1) = substr(addph,4,1) THEN disadd = disadd + 1 substr(addhc,3,1) = substr(addph,3,1) substr(addhc,4,1) = substr(addph,4,1) substr(addhc,5,1) = substr(addph,5,1) THEN disadd = disadd + 1 proc sort by last disadd disnam disnam2 data temp.add set temp.hcphbas by last disadd disnam disnam2 if last.last PROC APPEND BASE = temp.hcphb&b DATA = temp.add %end %mend banger %banger %END %MEND SAMPLE %SAMPLE EVALUATION OF THE METHODS At this point the reader can see, if you are still with me, that a set of descriptive, inferential, and graphical representations can be run to assess the nature of merging that occurred. DATA EAS.SAMPLE2 SET TEMP.HCPHB1 TEMP.HCPHB2 TEMP.HCPHB3 TEMP.HCPHB4 TEMP.HCPHB5 TEMP.HCPHB6 TEMP.HCPHB7 TEMP.HCPHB8 TEMP.HCPHB9 TEMP.HCPHB10 TEMP.HCPHB11 TEMP.HCPHB13 TEMP.HCPHB14 TEMP.HCPHB15 TEMP.HCPHB16 TEMP.HCPHB17 intage = int(age) CHANCE = 0 IF MATCH = 1 THEN DO CHANCE = 0.5 * ((DISADD/CUTLHC) + (DISNAM+DISNAM2)) END PROC SORT BY CUTLHC DESCENDING CHANCE PROC FREQ TITLE3 "FIRST HCFA - PHONEBOOK MERGE" TABLES MATCH TABLES MATCH*DISADD*DISNAM TABLES AGE80_01*MATCH / EXPECTED CHISQ TABLES SEX*MATCH / EXPECTED CHISQ TABLES RACE2*MATCH / EXPECTED CHISQ format match mat. age80_01 age. SEX SEX. RACE2 RAC. PROC TTEST CLASS MATCH VAR AGE format match mat. age80_01 age. SEX SEX. RACE2 RAC. PROC TTEST CLASS SEX VAR CHANCE format match mat. age80_01 age. SEX SEX. RACE2 RAC. PROC TTEST CLASS AGE80_01 VAR CHANCE format match mat. age80_01 age. SEX SEX. RACE2 RAC. PROC UNIVARIATE PLOT NORMAL VAR CHANCE AGE proc print VAR CHANCE DISADD DISNAM DISNAM2 NAMEH NAMEP ADDHC ADDPH PHONE BY CUTLHC RUN *LIBNAME EAS E:\DATA\EAS\HCFA98\DATA\ *LIBNAME TEMP E:\TEMP\ LIBNAME EAS LIBNAME TEMP D:\AECOM\EAS\HCFA98\DATA\ F:\TEMP\ *GOPTIONS DEVICE=WIN GACCESS= SASGASTD>LPT1: ROTATE=LSCAPE INTERPOL=JOIN GOPTIONS DEVICE=HPLJ4SI GACCESS= SASGASTD>LPT1: ROTATE=LSCAPE INTERPOL=JOIN DATA EAS.SAMPLE2 SET EAS.SAMPLE2 PROC FORMAT VALUE DEA 0= Alive 1= Dead VALUE AGE 0= <80 1= 80+ VALUE SEX 1= Male 2= Female VALUE RAC 1= White 2= Black 3= Other VALUE MAT 0= No Match 1= Match PROC FREQ TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Age Distribution TABLES INTAGE PROC univariate plot TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Merge Confidence var chance pattern1 value=solid axis1 color=blue width=2.0 axis2 color=blue width=2.0

8 axis3 color=blue width=2.0 proc gchart TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Overall Age Distribution vbar INTAGE / maxis=axis1 raxis=axis2 DISCRETE type=freq WHERE INTAGE > 65 proc gchart TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Age Density by Sex vbar INTAGE / maxis=axis1 raxis=axis2 DISCRETE type=pct by SEX format sex sex. where intage > 65 symbol1 c=default proc gplot TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Merge Confidence plot CHANCE * INTAGE WHERE INTAGE > 65 symbol1 c=default symbol2 c=default l=34 proc gplot TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Merge Confidence for each Sex plot CHANCE * INTAGE = SEX FORMAT SEX SEX. WHERE INTAGE > 65 CONCLUSION Animation presents itself as a powerful and necessary TRADEMARKS SAS and SAS/MACRO are trademarks of the SAS Institute Inc in the USA and other countries. indicates USA registration. REFERENCES Eckholdt, H. (1999). Visual Analysis on the WEB: Animating High Density Multidimensional Data. In M. Zdeb (Ed.) Proceedings of the North East SAS Users Group Washington, D.C., Eckholdt, H., Brown, L., Smith, D. and Feldman, S. (1998). Revealing Structure-Function patterns in the Basal Ganglia: Animating autoradiographic maps. Paper presented at the Society for Neuroscience. November 1998: Los Angeles, California. SAS. (1998). SAS for Windows version [computer software] Cary, North Carolina: SAS Institute, Inc. inquiries (919) ACKNOWLEDGMENTS The following researchers (great scientists, terrible technical managers) have provided me with data and inspiration that serve as the basis of this presentation: Lucy Brown, PhD, Herman Buschke, MD, Richard Lipton, MD, MPH. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Haftan Eckholdt, PhD, MS Albert Einstein College of Medicine Department of Neurology Kennedy Center Pelham Parkway South Bronx, New York Work Phone: (718) Fax: (718) eckholdt@aecom.yu.edu symbol1 c=default L=1 symbol2 c=default l=15 symbol3 c=default l=25 proc gplot TITLE2 Einstein Aging Study for Richard Lipton TITLE3 HCFA-Phone Merge Confidence for each Race plot CHANCE * INTAGE = RACE2 WHERE INTAGE > 65 FORMAT RACE2 RAC. run quit RUN