Building an Automated Data Management, Analysis and Reporting Tool That Runs Against Pre-Processed Web Server Access Log File Data.

Transcription

1 Building an Automated Data Management, Analysis and Reporting Tool That Runs Against Pre-Processed Web Server Access Log File Data. Nick Paszty, Berkeley California Abstract Web server access log files present a challenging data source due to tbe variety of server formatting configurations, frequent lack of unique identification as well as tbe sheer size of tbe files themselves. Using a home grown Perl pre-processor, logs are reduced to more manageable files containing standard reports in a standard format. This paper will demonstrate how we used the ODS in SAS version 8.x along witb Access to PC File Formats, Base, Graph/Annotate and Macro on Windows NT to manage web site data and produce drillable reports from the preprocessed data. The challenges of designing a scaleable standardized directory structure, learning tbe ODS and automating tbe process will be discussed. Introduction The purpose of this tool is to provide a library of programs tbat generate standardized data strucblres and reports in support of ad hoc analysis and insightful web site activity interpretations. The programs are written so that they can potentially be incorporated into AF application development. The development process has the following main challenges. 1. Building a standardized directory structure. When defining and developing new data structures it is good practice to design a standardized scaleable directory strucblre in which to store your files. In addition to data storage considerations, standardized web server directories are also necessary to manage H1ML content generated by tbeods. 2. Understanding log file data. This is a crucial and somewhat obvious exercise- know your data. A discussion of how to analyze web server access log file data and what to look out for is beyond the scope of this paper. There is however lots of information available for tbis topic on, you guessed it, tbe Internet. 3. Understanding the Output Delivery System (ODS). The ODS can be used to apply a standard format to all of your H1ML output. Working witb proc template and its many options while designing a company template for example should give you a descent understanding of how and where tbe ODS gets its design directives from. Some of tbe more difficult manipulations when using tbe ODS are coordinating tbe SAS Graph procedures and managing tbe resulting H1ML and image files. 4. Automating the process. Implementing a standardized directory strucblre and having a standard data source is a solid foundation on which to build automated processes. In addition to this, globally used file templates and catalog entries need to be in place so that once tbe process is automated, it has a rich library of support files tbat can be accessed. The remaining task is to set up tbe web site's data structure environment configuration file tbat contains all the necessary macro variables used during data processing and reporting. Retrieving Log Files The desired date range of pre-processed files is retrieved from a data-staging server using an FI'P client. A future enhancement could be to use SAS and the ftp argument on the filename statement. Data Description The data are stored in a one day per file structure witb around 50 sections. Each file's name is the web server access log file date in yyyymmdd format. Data transfers can include several weeks to several months of log files, which can mean tens upon tens of files to parse. Challenge 1. Standard directory structures For simplicity and efficiency, I use a Perl script to build data and H1ML content strucblres as well as copy critical template files into place. Tools There is usually a suite of utilities and common programs that can be used across all of your directory strucblres. An example of a utility program is one tbat builds the standardized directory strucblre for each web site's data and an example of a common program would be one that creates a critical success factor background annotation for a particular measure. The following template programs are copied from the tools directory into tbe appropriate directories under the new web site's main directory. ODS style program User defined formats program Web site environment configuration program Data Type Though this paper discusses programming against web server access log data only, I plan to have other data sources so tbe design of the directory structure must be flexible. The structure below all data_type level directories is standardized. So from the top down, you see something like this. \root \website \data_type \uber \fmts \root \tools \peri \tlb \ods \uber \doc \log \raw \sds \sds \sds Documentation files SAS log files SAS program files Raw data files SAS data set files Template program Formats program Formats catalog Traffic light programs Annotate catalog and data sets ODS style template program Environment configuration program 446

2 A nice advantage to standardizing your directories is that you only need to define two macro variables to access all of a web site's data, the web site's name and the data_type name. Web Server If you use the ODS features, you need to know a little something about web servers- among other things, the location of your server's document root This location is a directory that the web server sees as its top-level directory and is usually where you would find your home page file, commonly named index.html. To store the many reports, standard directories are also necessary under the web server's document root. These are set up to store html output generated by the ODS. \root \htdocs \website \htm \img \pdf Hyper text files Image files Acrobat files Challenge 2. Understanding Log File Data See Internet. Challenge 3. Understanding the Output Delivery System My first step in producing web enabled reports using the ODS is to define a style that can be applied to the entire set of HTML reports for a given web site. ODS styles are built using proc template and by modifying one of the SAS styles provided as part of the standard install, custom styles can be developed. The partial proc template code below illustrates how to inherit definitions from an existing style and set some custom style definitions. Here, the background and header image is set for all HTML output that uses this style. The macro variables are initialized in the web site's environment configuration file described in the Challenge 4. - Automating the Process section. The relative reference to the background GIF file points to two directories above the standardized structure under my web server's document root where globally used image files are kept proc template; define style &wsnm.. &wsnm; parent=styles.default; style Body from Body I backgroundimage='.. /.. /bg_image.gif preimage='.. /&wsheader gif watermark= true *...many more design options-.; end; This creates a style named the web site name under a directory named the web site name in the template store sasuser.templat Challenge 4. Automating the Process Several steps have to be taken to set up globally used files in support of automating the reporting process. Some of the globally used files include graphics catalog members that contain the traffic lighting backgrounds. These can be used for all measures that share the same critical success boundaries. For example I use a three tiered success characteristic, green-yellowred, for measuring site performance by way of entrance pages. Percentages are used to denote the color boundaries for our traffic lights and we use the annotate facility to produce a permanently stored background catalog member. The following code produces a globally used catalog member that is stored under the \root\tools\tlb\sds directory structure. libname tlb '\root\tools\tlb\sds'; title; footnote; * set traffic light critical values; %let green=60; %let yellow=so; goptions reset=global cback=white colors=(white); data tlb.tl_entrance_pages; length function style color $ 8; retain xsys ysys hsys '3'; * attributes for critical factor - green; function='move'; x=l2; y=17.4; output; function='bar'; x=ll2+((&green/10)*8.6)); y=90; color='cx006600'; line=o; style='s'; output; * similar code for yellow and red; * Clear all graphs from the WORK.GSEG catalog; proc greplay igout=tlb.tl_cat nofs; delete entpages; proc ganno annotate=tlb.tl_entrance_pages lighting ; gout=tlb. tl_cat name= entpages description='entrance pages traffic With a full compliment of backgrounds in place, the modifications to the three template files copied over by the directory building script should be made. Since I've covered the proc template code and user defined formats are straightforward, I'll describe some of the definitions and functions in the environment configuration file. Environment Configuration File (ECF) This file contains macro variable definitions for alllibrefs and an indexing counter used to facilitate data management and segment analysis by month and week. It also builds two control files that direct the parsing loop in the feeder programs, calls the feeder programs, initializes and concludes authoring the HTML table of contents links that control navigation through the reports and Excel files. Associations The goal was to only have to define one value that propagates throughout the code. Under a standardized directory structure, this strategy makes it easy to reuse the code for any web site. There are many Iibrefs and other associations defined in the ECF so I will only go over a few which should be adequate to illustrate the pattern of implementation. 447

3 Options errorabend noxwait fmtsearch=(userfmts work) source2; * macro var assignment for web site name; %let wsnm=paszty; * macro var assignment for doc root dir; %let docroot=\root\htdocs; * macro var assgnmnt for web site main dir; %let wsdir=\root\&wsnm; * macro var assignment for HTML files dir; %let webs_htm=&docroot\&wsnm\htm; * macro var assignment for raw files dir; %let webs_raw=&wsdir\webs\raw; * macro var assignment for data sets dir; %let webs_sds=&wsdir\webs\sds; libname &wsnm ~&webs_sds ; Control flies and counters It is necessary to know how many new pre-processed log files to expect to add to the existing data structures each time the programs are run. ECF has to create a counter to control looping and it and the subsequent feeder programs need to know if they have been run before for a given web site. * delete existing control file; x 'del &webs_raw\ppt_files.txt ; * build control file using dir command options bare and attribute (on NT); x 'dir /B /A:A &webs_raw > &webs_raw\ppt_files.txt ; data &wsnm.. lcntll (keep=filenm label='ppt extract primary control file'); * control file location and name; infile '&webs_raw\ppt_files.txt truncover; input filenm $8.; * include control line if it starts with '20' for yyyy being 20xx; if ('20'=substr(filenm,l,2)); * find number of observations in control file and create index macro variable; dsid=open('&wsnm.. lcntll'); call symput('index',trim(left(put(attrn(dsid,'no bs'),8.)))); dsid=close(dsid); The control file data look like this. FileName Calling Authoring and Feeder Programs The last action of the BCF is to call the programs that do all the work. This is done simply by using the %include statement. * start authoring the contents file; %include '\root\uber\contents_header.sas'; Here is the beginning of the table of contents HTML page authoring code, contents_header.sas. file '&docroot\&wsnm\htm\contents.htm'; put '<html>' ; put '<head>'; put '<title>table of Contents</title>'; put '</head>'; put ' '; put '<body leftmargin=o bgcolor='#ff9900' vlink='# link='#0066aa'>'; * more tags here; * include all feeder programs; %include '&webs_pgm\feeders.sas ; * stop authoring the contents file; %include '\root\uber\contents_footer.sas'; Feeder Programs These programs do all the work of parsing each of the desired pre-processed file sections into variables and creating data sets with chronological month and week indexes. They also create the critical success factor reports, Excel spreadsheets and contribute their particular section report hyperlinks to the table of contents HTML page. Bach of the feeders does the same general type of parsing, data management and reporting so I will describe what I think are the main programming concepts used to accomplish these tasks. Parsing Each of the 50 or so sections of the pre-processed data can be parsed. The key is to identify the beginning of the section to be parsed, and the beginning of the next section. The next step is to be able to parse the same section from one report file after another and aggregate the data into a final temporary data set. The process runs inside a macro whose arguments are the first and next section names. The macro also makes use of the control file and index counter macro variable generated by the BCF. Here are the main pieces of the entrance page report parsing code. %macro parser(startxt=, endtxt=); %* index macro variable is assigned its value in ECF file and controls looping through all data files; %do i=l %to &index; %* assigns value to macro variable by looping through primary control file; 448

4 set &wsnm. lcntl1; if _n_=&i; call symput('filenm',trim(left(filenm))); t* data step finds first and last line of data file to parse; infile '&webs_raw\&filenm truncover; input logline $500.; t* first line of entrance page report; if logline= &startxt then do; call symput ( 'startn', n_); end; %* last line of entrance page report; if logline='&endtxt then do; call symput ( 'endn', n_) ; end; t* parses each line into its constituent parts; data entrance (keep=nhits entrance_page logdtl; length nhits 4 entrance_page $200; t* first and last observations only as assigned in _null_ above; infile '&webs_raw\&filenm truncover firstobs=&startn obs=&endn; %* grab entire line; input logline $500.; %* create SAS date value; logdt=input('&filenm',yymmdd8.);...more parsing code...; t* loops that append each newly created data file to a final data file; %if &i=l %then %do; data fnl_entrance; set entrance; \end; %else %do; data fnl_entrance; set fnl_entrance entrance; %end; %* ends &index do loop; %end; %mend parser; %parser (startxt=entrance Pages, endtxt=all Response Codes) ; Data Management The next step is to aggregate the data and build the month and week indexes that are used by the reporting code. For simplicity, I decide to rebuild all the chronology indexes after appending new data to existing data. The process runs ioside a macro where a check for existing data is done using the exist() function. First I add three chronology variables to the fnl_entrance page data set that are used later to build continuous chronology indexes. * retrieve date parts from logdt variable; yr=year(logdt); mn=month(logdtl; wkday=weekday(logdt); Here are the main pieces of the data management code inside the exist() macro code. %macro dsexists; %if %sysfunc(exist(&wsnm.. entrance_pages)) %then %do; %* create data set containing all records from parse loop; data entrance_pages; %* drop existing indexes to ensure proper re-build in next data step; set &wsnm.. entrance_pages (drop=mnindex wkindexl fnl_entrance; %* sort data set to ensure proper initialization of first. and last. variables; proc sort data=entrance_pages; by yr mn logdt; %* create permanent data set containing all records from parse loop; data &wsnm.. entrance_pages (label='entrance pages'); set entrance_pages; by yr mn logdt; t* retains value across records; retain mnindex wkindex; t* assign month indexing value at first record; if n_=l then mnindex=l; %* month index counts months from 1 to n; else if first.mn then mnindex=mnindex+l; %* assign week indexing value at first record; if n_=l then wkindex=l; %* week index counts weeks from 1 to n; else if (first.logdt and wkday=7) then wkindex=wkindex+l; %* label new variable; label mnindex='month number' wkindex='week number'; 449

5 %* ends If data exists loop; \end; \else %do; _code if first time web site data are ~"; \end; \mend dsexists; Reporting The reporting code accomplishes five tasks. I. Builds reports & drill downs for the total time range 2. Builds hi-low reports & drill downs across total date range 3. Builds reports & drill downs by month 4. Builds reports & drill downs by week 5. Outputs by week data to an Excel spread sheet Because of limitations, there are a number of data management steps that I am not going to describe here. Take the data structures for granted and imagine using proc means to output data sets where the sum of hits aggregated by entrance page is stored by the chronological indexing variables with a top_n records identifier. I will not describe the hi-low process either because it is essentially the same as the process used to develop reports by month with the slight distinction of having two bars on the chart rather than one. I have chosen to describe the reporting process using the by month code which is similar to the by week code in almost all respects. To control the reporting loops, I create a number of macro variables that contain values for events like first month and last month. The values are calculated from the permanent entrance_pages data set by identifying the month index at the first and last observation. To control the number of observations included in the report, I use a macro variable, top_n, whose value is set at the top of the program. This makes it easier to modify globally if the desired top_n observations need to be decreased or increased. To control the critical factor ranges, I use two macro variables, yellow and red, whose value is also set at the top of the program. Currently, these need to be the same range values used to create the traffic light backgrounds described above. Step 1. A master macro is needed that can loop through an aggregated data set created by proc means one chronological index at a time. This macro calls another macro that does the reporting for the current index. The code below is part of the master macro. %* start to loop through all months by way of month index; %do i=l %to &mnindex; %* use data null to create macro variables for each iteration; if _n_=l; set mnhits (where=(mnindex=&i)); * for report titles; call symput( year',yr); call symput('month',put(mn,mn.)); %* call macro mns; %mns(&i, &year, &month); %* lots more code; Step 2. A month reporting macro is needed to build the drill down hyperlink. the indicator bar, the critical factor values and the critical factor macro variable used to control charting. %macro mns (mn, year, month); %* build month data sets; data mnhits&mn (keep=entrance_page detail_drill std_hgt yr mn mnindex top_n mnhits&mn percent critical_20 critical_lo critical_ol; %* assign length to bookmark link - problems if too short!; length detail_drill $100; %* restrict observations to month index and top_n entrance pages; set mnhits (where=(mnindex=&mn and top_n<=&top_n) ) ; %* assign value to book mark link - each month is stored in a htm file; detail_drill="href='entrance_pages~onth&mn.. htm#detaill'"; %* creates indicator bar for chart; std_hgt=25; %* calculate percent of total hits within month; percent=round((mnhits/&monthobs)*loo); %* retain value from one iterations of the data step to the next; retain critical_20 critical_lo critical_o; %* red light; if (_n_=l and percent>&red) then critical_20=percent; else if (_n_>l and percent>&red) then critical_20=critical_20; %* similar code for yellow and green; %* rename sum of hits variable for each month; rename mnhits=mnhits&mn; %* label new variables; label mnhits='month &mn - &month, &year &month &year ; percent='percent month &mn - %* outputs only last observation to data set mnhits to get the highest critical value; data mnpcnt&mn; if lastobs; set mnhits&mn end=lastobs; %* create macro variable for conditional charting; 450

6 set mnpcnt«mn (keep=critical_20 critical_lo critical_o); t* create critical level macro variables; if critical_20=. then call symput('c_20',trim(left(put(0,8.)))); else call symput('c_20',trim(left(put(critical_20,8.) ) )) ; t* similar code for yellow and green; t* statements used to re-direct graphics output; filename gifout &webs-ptm\entrance_pages~onth«ron.. gif ; goptions gsfname=gifout; t* titles and footnotes place on GIF file; titlel "Entrance Pages Percent From One Source- &month &year ; footnote! j=l h=2 'Entrance Pages Performance: c=cx 'Leader c=cxcc9933 'Caution ' c=cxffoooo 'Problem' j=r c=cxoooooo 'Click on bar to view detail'; t* define macro to do conditional charting; tmacro critfact; t* graphics procedure to produce the histogram indicating critical level; proc gchart data=mnpcnt«mn gout=gseg; t* will create chart only if critical value was 'red'; tif &c_20 tthen %do; vbar critical_20 I tend; sumvar=std_hgt midpoints=o to 100 by 5 maxis=axisl raxis=axis2 html=detail_drill width=.s no frame; -more loops for yellow and green-; t* start ods output with web site style and referencing the bookmark link; ods html body= &webs-ptm\entrance_pages~nth&mn.. ht m style=odstyle.&wsnm anchor='detail'; t* moves the entrance pages traffic lighting background to the default work catalog; proc catalog cat=tlb.tl_cat; copy out=gseg; select entpages I et=grseg; t* Use PROC GREPLAY to overlay the background and the chart; proc greplay igout=gseg nofs gout=gseg tc=sashelp.templt template=whole; treplay l:gchart&mn l:entpages; Step 3. The last step is to build a data set that is exported to Excel, run the proc dbload code, delete all the members in the work library and include the table of contents authoring code. The code below merges any number of data sets so that all the by week data can be loaded into Excel. * macro that merges all week data sets; %macro mergewk; data allnhits (drop=top_n detail_drill std_hgt percent critical_20 critical_lo critical_o yr mn mnindex); wnhits&i) merge allnhits by top_n; tmend mergewk; %do i=l tto &wkindex; tend; wnhits&i (keep=top_n * ends merge; The authoring file uses the month indexing macro variables to create the month hyperlinks for this report section. tdo i=l tto &mnindex; mnt=put(&i,mn.); put '<font face="arial, Helvetica, Helv" size="2" color="#oooooo ><dt><b> <lb>'; put '<A HREF="entrance_pages_month'"&i''.htm' TARGET='body'>'mnt'<la>'; tend; Conclusion This tool relies on SAS version 8.x ODS features to facilitate development of web ready reports. Building a standardized directory structure produces an ad hoc analysis friendly database for each web site's data and establishing automated mechanisms pulls it all together with a simple double-click. References SAS OnlineDoc, Version Eight, Copyright (c) 1999 SAS Institute Inc., Cary, NC, USA. All rights reserved. Acknowledgements I would like to thank EMITS and all SAS-Lers who responded to my program development related questions. You can contact me at: ngp@paszty.com SAS and SAS/ ACCESS are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. <II> indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies. 451