Re-programming a many-to-many merge with Hash Objects

Transcription

1 Re-programming a many-to-many merge with Hash Objets CS05 PhUSE 2012 Budapest D. J. Garbutt 17 Otober 2012

2 Why? 1 In this talk I will demonstrate how you an re-program a many-to-many merge using hash objets instead of the various other methods already known. to illustrate the advantages of hash objet for typial SAS programming

3 What? Two worked examples in the paper and on Wiki page 1 amanytomanymerge CMbakintoAE 2 acdiscdatavalidation hekvoabularies aross domains and variables One data step with no maro variables at all. In this presentation I will look at just the seond example whih omes from a paper at PhUSE last year by Guido Wendland. Chek it out.

4 How and When to use a hash objet How? Tips and syntax details See the paper. lots of further reading When? Hashes an be imported into any data step and an be used in many instanes where maro variables would be normally be needed. Hashes are not just for look up of large tables

5 Up Next 1 Introdution and History 2 Chek a CDISC Domain for invalid values The problem The data The programming Adding other variables Adding other datasets 3 SAS programming after the hash objet SAS Memory model Hash Objet myths Issues with SAS program development Solved!

6 Hash arrays and Content Addressable Arrays 1977 AWK and Snobol4 in 1977 Currently in KornShell 93, bash, Ruby, PowerShell,.NET, Java, R, et., Dorfman showed hashing ould be programmed with adatastepinthe90 sbutitisnotsimpleandonly worth the effort for very large datasets. Now sine 2008 (9.2) SAS has built-in support for the hash objet.

7 AWK program 1 xargs file awk { $1=NULL; t[$0]++; } END { for (i in t) printf("%d\t%s\n", t[i], i) } sort -nr

8 How does the hash array look? name[ Dave ] + 1 ; index: [Dave][dave][Greg][Urs][n/a] address: Count: An Array: is a storage area and arule: Alloate area of ontiguous memory then to aess it alulate storage loation from the index.

9 How does it work? 1 normal array rule: take the memory base address and add index (bytes in integer) hash array rule: take the memory base address, now run index = harater string through hash funtion to get an integer, add it to base address.

10 How does it work? 2 AhashfuntionisabitliketheMD-5keyorrandom number generator it does bit operations on the haraters, in this ase to get an integer.

11 How does it work? 3 Ahasharrayisnotalistdatastruture It has no links between members and annot grow indefinately. And insertion / deletion operations take onstant time O(1).

12 The problem Chek variables for orret values Some SDTM variables (data elements) have fixed ontent so we would like to hek and make sure they onform. Ingredients 1 Data domain e.g. AE 2 list of variables to hek with the name of their ode-list 3 SDTM ontrolled data list SDTM/SDTM/Terminology.odm.xml

13 The problem Solution with maro Asolutionwasverylearlypresentedatlastyear s PhUSE by Guido Wendland ASASmarotohekSDTMdomainsagainst ontrolled terminology, PhUSE 2011,

14 output ; end; /* next test follows...*/ The problem Generate a data step fragment like data violations (keep=variable error) ; set ae ; if aean not in ( DOSE INCREASED, DOSE NOT CHANGED, DOSE REDUCED,...) then do; variable = AEACN ; error = ats( value of AEACN,aean, not found in ditionary ) ;

15 The problem To get this ode Transpose the ode list then substitute it into an in() test Repeat for eah variable. Thousands of lines of ode for SAS to interpret and exeute

16 The data flow

17 The data The goal is to hek values 1 data violations (keep=variable error) ; set ae ;

18 The data The goal is to hek values 2 if aean not in( DOSE INCREASED, DOSE NOT CHANGED, DOSE REDUCED,...) then do; variable = AEACN ; error = ats( value of AEACN,aean, not found in ditionary ) ; output ; end;

19 The data The goal is to hek values 3 run; /* next test follows...*/

20 The data The goal is to hek values 4 Could we do a fast lookup of the right ode? We would put all the voabulary in one hash objet and use the hek() method to test for presene

21 The data Something like this: 1 notfound = odelst.hek( key: CODELIST_CODE, Key: AEACN );

22 The data Something like this: 2 if notfound then do; variable = AEACN ; error = ats( Value of AEACN,aean, not found in ditionary ); output ; /* next variable... */

23 The data Something like this: 3 Notes: no quotes round AECN We need the ode lists...

24 NCI Code lists Code Code List Code Codelist Name CDISC Subm C66767 Ation Taken with Study Treatment ACN C49503 C66767 Ation Taken with Study Treatment DOSE INCRE C49504 C66767 Ation Taken with Study Treatment DOSE NOT C49505 C66767 Ation Taken with Study Treatment DOSE REDU C49501 C66767 Ation Taken with Study Treatment DRUG INTE C49502 C66767 Ation Taken with Study Treatment DRUG WITH C48660 C66767 Ation Taken with Study Treatment NOT APPLIC

25 The data Split into two datasets read the XML file filename SDTMCT H:\garbutt\DATA\Projets\CS05-djg\SDTM_ filename SXLEMAP H:\garbutt\DATA\Projets\CS05-djg\CDISC libname SDTMCT xml xmlmap=sxlemap aess=readonly;

26 The data manipulate the datasets /* the map defines two tables */ data odelist_m; SET SDTMCT.CodeList_M; run; data odelist; set sdtmt.odelist; odelist_name= san(oid,3,. ) ; run;

27 The data Read the ontrol dataset maplist data as used by Wendland data s05l.maplist ; length ruleid $6 variable $8 Codelist $411 ; input odelist $10. ; datalines ; CT0001 AEACN ACN CT0002 AESEV AESEV CT0009 DOMAIN DOMAIN CT0027 AEOUT OUT CT0037 AEBODSYS SOC CT0064 AESER NY ;

28 The programming Setup the hash objets; Lookup the data varleft = vars_to_hek.first(); r = odelist_md.find(); notfound = odelst.hek( key: CODELIST_CODE, Key: AEACN );

29 Program flow and PDV ontents

30 Adding other variables Iterating through the datasets add hash iterator to the maplist hash objet /* Maplist mathes variable and odelist (num id) */ delare hash maplist(dataset: "s05l.maplist", hashexp:4, ordered: yes ); r + maplist.definekey( VARIABLE ) ; r + maplist.definedata( VARIABLE, RULEID, CODELIST ); r + maplist.definedone(); all missing(variable, ruleid, odelist) ; if r then putlog ERRO R: problem reating hash obj maplist ; /* define iterator for maplist to enable looping */ delare hiter vars_to_hek( maplist ) ;

31 Adding other variables Iterating with first() and next() varleft = vars_to_hek.first(); do while (varleft = 0 ) ; RC = 0; notfound = 1 ; all missing(ruleid,,odelist,odelist_value,odelist_ode ) ; varleft = vars_to_hek.next() ; end;

32 Adding other variables Lookup the variable value to hek valtotest = vvaluex(variable) ; notfound = odelst.hek( key: CODELIST_CODE, Key: valtotest );

33 Adding other datasets Looping aross datasets 1 Compile the data step and generate with all exeute (1 of 4 ways) *--- ompile program ---- ; data violations /PGM=work.violations ; run; *--- and generate list of members in vmembers from maplist--;

34 Adding other datasets Looping aross datasets 2 Compile the data step and generate with all exeute (1 of 4 ways) data _null_ ; set vmembers ; all exeute ( data PGM=work.violations ; ) ; all exeute( exeute ; ) ;

35 Adding other datasets Looping aross datasets 3 Compile the data step and generate with all exeute (1 of 4 ways) all exeute( att( rediret INPUT AE=, memname, ); ) ;

36 Adding other datasets Looping aross datasets 4 Compile the data step and generate with all exeute (1 of 4 ways) all exeute( att( rediret output violations =v, memname, ; run; ) ); run;

37 Adding other datasets Full ode is about 120 lines 50 lines of hash objet delaration not a single maro variable headed for the wiki...

38 SAS memory model Till now

39 SAS memory model After hash objets

40 Hash Objet myths 8 Hash Objet myths dispelled 1 1 They are just for lookup with millions of rows I don t need that syntax mess! 2 Useless you an t have dupliate keys! multidata: yes 3 If keys are dupliated you get no warning! dupliate: error

41 Hash Objet myths 8 Hash Objet myths dispelled 2 4 You an t sort them! ordered: yes 5 You ll run out of memory! 4GBis5.3million800bytereords 6 You have to build the hash with add(), delete() et. Tedious! dataset: work.dave (where myths > 9)

42 Hash Objet myths 8 Hash Objet myths dispelled 3 7 Can t be used outside a data step! output() method will write to dataset 8 Merge, Pro SQL, set key=, and formats work fine for merging hash is faster and advantage inreases with size of data

43 Hash Objet myths 8 Hash Objet myths dispelled Syntax is verbose and opaque 11 Can t be used with pros, maros, the DDV or pro IML

44 Issues with SAS program development Fundamental Issues with SAS program development 1 Longstanding issues sine Version 6 and earlier Why is developing large sale systems so hard with SAS? My list of reasons is: 1 separation of pros and data steps from eah other by the disk transfer wall

45 Issues with SAS program development Fundamental Issues with SAS program development 2 Longstanding issues sine Version 6 and earlier 2 maro is purely a text generator (f Korn Shell, Perl, Ruby) 3 maro variables are simple, individual strutures. Other strutures have to be emulated by naming onventions and ollision avoidane. Or not.

46 Issues with SAS program development Fundamental Issues with SAS program development 3 Longstanding issues sine Version 6 and earlier 4 in data steps only salars are allowed (ie single named elements of PDV). Arrays are just list of variable names, not data strutures! 5 lak of truly enapsulated funtions or subroutines (goto label: are you kidding me?).

47 Issues with SAS program development Fundamental Issues with SAS program development 4 Longstanding issues sine Version 6 and earlier 6 lak of an exe() funtion that an exeute ode in plae and return the result

48 Solved! Issues with SAS program development 1 solved! In Version 9.2 SAS programming has jumped into the present and the previous issues addressed: Hash Objet adds matrix objets to data step, with fast searh, size independent performane and flexible data types

49 Solved! Issues with SAS program development 2 solved! Pro FCMP allows funtions to enapsulate user ode and for it to be used as normal SAS funtion. It supports arrays and all by referene. run_maro, dosub & dosubl allow data steps and pros to be alled from a data step and pass results bak via maro variables

50 Solved! Issues with SAS program development 3 solved! JSeosky,ExeutingaPROCfromaDATAstep, SAS GF 2012, proeedings12/ pdf

51 Solved! The hope for meta data Meta data programming and meta data systems are the next wave of hange

52 Solved! The hash objet opens new doors shorter ode easy to debug (an use data step debugger) easier to maintain For designing and building systems, espeially based on metadata, these three new features will make an enormous differene to SAS programming and how it is done. It is the 21st entury, SAS programming is fun again datasets and data steps an drive proessing without maro

53 Solved! And the next doors: please SAS: 1 Allow hash objets to be passed to pro FCMP funtions 2 Allow hash objets in open ode and 3 to persist outside data steps 4 Allow pros to be passed hash objets as if datasets 5 Allow pros to write hash objets as output

54 Questions Questions?

55 PhUSE Wiki title=reprogramming_a_many-to-many_merge_ with_hash_objets Where you an also add ode, examples, and disussions. NOTE: Wath out for papers written before V9.2 Theymaynowbemisleading dupliatesareallowed, theymayalsohaveworkaroundsthatareno longer needed. Questions Referenes, further reading