Adapting AWStats for IBM WebSphere Portal 6.0.x and virtual portals

Transcription

1 Adapting AWStats for IBM WebSphere Portal 6.0.x and virtual portals Markus Moltenbrey IT Specialist, IBM Software Group IBM Deutschland GmbH August 2008 Copyright International Business Machines Corporation All rights reserved. Abstract: The functionality to track site usage is still provided as of IBM WebSphere Portal version 6.0.x, but the Tivoli Web Site Analyzer that charts the resulting log files from version 5.1 was deprecated. As a result, the open-source solution AWStats has become increasingly popular with WebSphere Portal customers. The free software product creates extensive charts formatted for browser access with minimal effort; however, it cannot discern between virtual portals and does not display the humanreadable part of the site URL string. This white paper explains how to extend AWStats to provide the missing functionality.

2 Contents 1 Introduction Colors & naming conventions Process overview Site analyzer Prerequisites Installing perl Installing Apache Installing Java Runtime Environment (JRE) AWStats installation and configuration Installation Configuration Pre-AWStats script installation Script description Installation URLs and tasks Script configuration Cronjob configuration The script package variable.sh awstats_update.sh prepare_parsing.sh parsesalog.pl exprequest.xml transform.xsl AWStats sample configuration (truncated) Configuration alternatives Non-Portal-Server hosting AWStats Anonymous usernames XMLAccess export Conclusion About the author Resources...23

3 1 Introduction Many customers require a basic site analysis of their IBM WebSphere Portal environment. Here we examine a cost-effective solution combining the integrated site analyzer engine--which provides information comparable to an Apache access.log --and AWStats, an open-source Perlbased application with which we can visualize the resulting protocols (see figure 1). Figure 1. AWStats example In WebSphere Portal 6.0 the log entries of accessed pages consist of long, cryptic strings in which the page title is shown at the very end of the request URL. The resulting graphs and statistical data aggregated by AWStats are not really useful because only the first parts of the strings are displayed in the graphs; the page title is not displayed (see figure 2).

4 Figure 2. AWStats output example for WebSphere Portal 6.0 Moreover, accesses to different virtual portal sites are logged in the same file and represented by one resulting graph. If the information is to be presented completely separate from each other and the accessed pages are to be human readable, then some adaptations are required. This white paper explains a solution that was implemented at one customer site. Note that it does not include detailed installation instructions for Apache, AWStats, or WebSphere Portal; for information on these topics, see the developerworks article, IBM WebSphere Developer Technical Journal: Using portal analytics with open-source reporting tools. 2 Colors & naming conventions The color codes we ll use in this white paper are as follows: User Input Variable name Directory Filename Code/Command (Example) Table 1 explains the site-specfic parameters we ll use. You ll need to replace them with your environment s site-specific values. Table 1. Our site-specific parameters $vp_name Virtual Portal URI Context, e.g. $awstats_install_root Installation directory of AWStats. Default: /usr/local/awstats $script_root Root directory where the provided scripts are located, e.g. /usr/local/scripts $portal_server_root Default product Location, e.g. C:\IBM\WebSphere\PortalServer $portal_admin_id WebSphere Portal Server Administrator User ID (Short) $portal_admin_pw WebSphere Portal Server Administrator User Password $portal_hostname The fully-qualified domain name of WebSphere Portal $wc_default_port WC_DEFAULT Port of WebSphere Portal Server Instance $portal_admin_id WebSphere Portal Server Administrator User ID (Short) $was_root Default product Location of WebSphere Application Server

5 3 Process overview Basically, nothing was changed in the source code of AWStats, though it would have been possible. To provide compatibility with later releases and to keep the process flexible, we implemented a kind of pre-processor and scripts to automatically adapt the site-analyzer log files and deliver better data to AWStats (see figure 3). Figure 3. Process overview In brief, the requirements for this customer s configuration are: 1. Completely separated statistical information for the several different virtual portal sites. 2. The displayed WebSphere Portal pages must be human readable. 3. The statistics must be updated frequently and automatically. 4. Users accessing WebSphere Portal must be tracked to deliver data on how many users accessed the site while keeping the users' names anonymous. 4 Site analyzer Site analysis is not enabled by default. To activate the data collection for WebSphere Portal, you must set the parameters shown in table 2 in the WP_SiteAnalyzerLogService. See the IBM WebSphere Portal information center for more information on how to set configuration properties.

6 Table 2. Parameter values in WP_SiteAnalyzerLogService Parameter Value SiteAnalyzerFileHandler.fileName log/sa_$appserver_name.log SiteAnalyzerFileHandler.backupFileName log/sa_$appserver_name_$crea TE_TIME.log SiteAnalyzerFileHandler.dateFormat yyyy.mm.dd-hh.mm.ss SiteAnalyzerFileHandler.daysPerLogFile 1 To activate selected loggings, set the corresponding values to true (see table 3). Table 3. Selected loggings SiteAnalyzerSessionLogger.isLogging SiteAnalyzerUserManagementLogger.isLogging SiteAnalyzerPageLogger.isLogging SiteAnalyzerPortletLogger.isLogging SiteAnalyzerPortletActionLogger.isLogging SiteAnalyzerErrorLogger.isLogging SiteAnalyzerApplicationActionLogger.isLogging true false true true false false false Here is an example log entry: Mark Mustermann [20/Jun/2008:04:36: ] "GET /Page/[ObjectIDImpl_'6_QRQMLMFAUFUJ902T20FMDT1I92'_[ : @- 1196_/_CONTENT_NODE],_Domain:_[Domain:_rel],_DB_representation:_54FB- 7A6B5BED53FEF99980E8023CDB7A9049]/News HTTP/1.1" "" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR ;.NET CLR ; InfoPath.1)" "JSESSIONID=00018zhK4eeMpyo6GJsB- UK0Zhq:- 3483ID;LtpaToken=pwTAXkGiuWs4MYE4FKKb7WCPfFnwThtigHGXlc/yASknHmWCKXp3v 4IDkNapj2Ws2y8fbqx/TwRqtF/KxUBoys9cWOU8GYSUhM+OUizUXFcWrGVwX11vMGZU/lv 66m64RjbX0znh85oWfFmK2LP7hG8G/cV8eINA0VdQvVVOw9FXkJ8qnyXoTOqNY+njGv1b4 mvl9ajagomr5pl5xtxav+numj10br+dsygpx3lirw0u4luaqjuu/kmearfgb3aa2kslpeu SapJBdvH4opq5FS0NsA0VHvMR+avG8BKh2Zsx/lHBLzu4oxndva7F9RsOBxmCrgv+jDBUQ 523LOR/mhkJ3uwC3rb3uj5/4+reZG0=" The following data can be derived from the log entry: The request was made from IP The authenticated user for this request was Mark Mustermann Handling the request was finished at 20/Jun/2008:04:36:23, GMT The page with title "News" was requested. The request was successful (HTTP response code 200). The size of the returned markup is unknown to the logger (-1). The request was made by a browser that identifies itself as "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR ;.NET CLR ; InfoPath.1)" The request was made within a session, identified by the id "00018zhK4eeMpyo6GJsB " After modification with the scripts, the log entry looks similar to the following:

7 $1$Ved3sWIq$DL/0e1WN35lumC2RfHUIb/ [20/Jun/2008:04:36: ] "GET /Page/(news.unique.name)/News HTTP/1.1" "" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR ;.NET CLR ; InfoPath.1)" "JSESSIONID=00018zhK4eeMpyo6GJsB-UK0Zhq:- 3483ID;LtpaToken=pwTAXkGiuWs4MYE4FKKb7WCPfFnwThtigHGXlc/yASknHmWCKXp3v 4IDkNapj2Ws2y8fbqx/TwRqtF/KxUBoys9cWOU8GYSUhM+OUizUXFcWrGVwX11vMGZU/lv 66m64RjbX0znh85oWfFmK2LP7hG8G/cV8eINA0VdQvVVOw9FXkJ8qnyXoTOqNY+njGv1b4 mvl9ajagomr5pl5xtxav+numj10br+dsygpx3lirw0u4luaqjuu/kmearfgb3aa2kslpeu SapJBdvH4opq5FS0NsA0VHvMR+avG8BKh2Zsx/lHBLzu4oxndva7F9RsOBxmCrgv+jDBUQ 523LOR/mhkJ3uwC3rb3uj5/4+reZG0=" Table 4 explains the adapted sample log entry in more detail. Table 4. Detail of the sample log entry Original String Mark Mustermann GET /Page/[ObjectIDImpl_'6_QRQMLMFAUFUJ902 T20FMDT1I92'_[- 1196_/_CONTENT_NODE],_Domain:_[Domain :_rel],_db_representation:_54fb- 7A6B5BED53FEF99980E8023CDB7A9049]/Ne ws HTTP/1.1 Replacement $1$Ved3sWIq$DL/0e1WN35lumC2Rf HUIb/ GET /Page/(news.unique.name)/News HTTP/1.1 5 Prerequisites To run AWStats, you must have a running Web server, for example, Apache, with enabled Perl module (mod_perl) to host the analyzer application. Also, besides an HTTP server, the provided scripts require a Linux/UNIX-based operating system and a Perl installation with the following additional modules (available from CPAN): XML:XPath XML::Parser 5.1 Installing perl If you are running a Linux system (or most UNIX systems, including Mac OS X), you probably already have an installation of perl that was packaged with it. Determine the version by issuing the following command: perl -v You should get an output similar to the following:

8 If perl is not installed on your current environment, install it using one of the following methods: Via precompiled packages of your Linux distribution (for example, RPM or DEB). The appropriate files should be located on the installation CDs or on one of the distributor's package servers. On SuSE try YAST, and on RedHat try YUM to install perl. For example, on RedHat, issue the following command as root user to install perl: yum install perl Download and install ActivePerl: d58c2648ca Download, compile, and install the perl source code from CPAN: Installation steps for additional perl modules: 1. Download the package from CPAN: wget Package XMLParser2.36.tar.gz XMLXPath1.13.tar.gz Download URL ANT/XMLParser2.36.tar.gz ANT/XMLXPath1.13.tar.gz 2. Extract the package on your server to a temporary directory: tar zxvf packagename.tar.gz 3. Switch to the package directory and issue the following commands to compile and install the module: cd package_directory perl Makefile.PL make make test su - make install 4. Repeat the above steps for each module.

9 5.2 Installing Apache Some distributions include a prepackaged version of Apache on the installation discs or package server. To install Apache via the distributor's specific packages, use either yum, yast, or aptget (depending on your distributor) to install Apache HTTP Server, including the required perl module extension. If you do not have the option of installing Apache by this method, or the version provided is not the desired on download, then install the Webserver according to the instructions on the Apache Web site. 5.3 Installing Java Runtime Environment (JRE) Either use the IBM JRE delivered with WebSphere Application Server located in $was_root/java or download an RPM package from either the Sun or the IBM developerworks Linux Download information Web site. 6 AWStats installation and configuration AWStats is available for download from the AWStats Official Web site. 6.1 Installation Since AWStats is a perl application, no compilation is required. Just download it, extract the downloaded package, move the content to /usr/local/awstats, and run the configuration wizard from inside /usr/local/awstats/tools: tar zxvf awstats-x.x.tar.gz -C /tmp/ mkdir /usr/local/awstats mv /tmp/awstats-x.x/* /usr/local/awstats cd /usr/local/awstats/tools./awstats_configure.pl Config Wizard Question Description Answer (example) Config file path ('none' to skip Web server setup): Provides the full path to your http /etc/httpd/conf/httpd.conf server's configuration file Do you want me to build a new AWStats config/profile? Answer this question with No since we will create the configuration profiles later In the Apache Webserver's configuration file the following changes should be visible: N

10 NOTE: If you do not want the wizard to adapt your httpd.conf, for example, if additional configurations are located in /etc/httpd/conf.d and are only included via the httpd.conf, you do not need to run the configuration wizard for installation. Instead, copy httpd_conf from $awstats_install_root/tools to /etc/httpd/conf.d/awstats.conf and adapt the paths within the file, if you installed AWStats to a nonstandard directory. 6.2 Configuration To create an AWStats configuration profile, just re-run the awstats configuration wizard (awstats_configure.pl). If you want to create one report per virtual portal server, you must create one configuration profile for each virtual server. Repeat the following steps for each profile you want to create: 1. Start the wizard with the following command within the AWStats installation directory: cd /usr/local/awstats perl /usr/local/awstats/tools/awstats_configure.pl 2. Answer the questions as shown in table 5: Table 5. Configuration wizard questions & answers Question Description Answer Config file path ('none' to skip Web server setup): Do you want me to build a new AWStats config/profile file (required if first install) [y/n]? Your Web site, virtual server, or profile name: Only specify a configuration file here if this is the first time your run the wizard. Answer this question with 'Yes' to start the creation of a new configuration profile Use the URI context name of your virtual portal(s) here instead of a fully qualified hostname, e.g. None 3. Now edit the configuration files (found in /etc/awstats) for each virtual portal, using the values shown in table 6. Table 6. Configuration file values Parameter Description Value LogFile LogType LogFormat DirData The directory where the log files that shall be analyzed are located. $vp_name must be replaced by the name of the corresponding virtual portal, e.g., VPx The type of log file that will be analyzed. ('W' means Web log file) The format of one log entry in the log file Only change this value if you do not want to place the AWStats in the default directory. (If the directory is not present, create it with 'mkdir /var/lib/awstats') Y VPx "/usr/local/awstats/salogs/$vp_ name/$vp_name.log" W "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot" "/var/lib/awstats"

11 7 Pre-AWStats script installation 7.1 Script description Table 7 lists the scripts necessary for installation and what each one does. Table 7. Script descriptions Script awstats_update.sh variables.sh prepare_parsing.sh parsesalog.pl exprequest.xml transform.xsl Description Main script; executes the other required processes. Stores required environment variables and properties, and is executed by awstats_update.sh - Invokes the xmlaccess task to export the portal site structure of each virtual portal. - Transforms the xml using transform.xsl. - Merges the site analyzer log files; backs up and deletes old files. - Invokes sub-scripts parsesalog.pl - Parses the original Site Analyzer log file and splits it into one file per virtual portal - Replaces the page URLs with a more human-readable version - Anonymizes the username component XML input file for xmlaccess request to export the page structure of the (virtual) portal XSL to transform exprequest.xml into a more simple form containing only AWStats-relevant information 7.2 Installation To use a script, extract the package and copy the files to a directory, for example, /usr/local/scripts, on your WebSphere Portal server (see table 8). Table 8. Pre-AWStats scripts and their default installation directory File Directory awstats_update.sh /usr/local/scripts prepare_parsing.sh /usr/local/scripts/awstats variables.sh /usr/local/scripts/awstats prasesalog.pl /usr/local/scripts/awstats exprequest.xml $awstats_install_root/salogs/ transform.xsl $awstats_install_root/salogs/ 7.3 URLs and tasks To manually update the AWStats configuration of the new sites, enter the following command at the command prompt, ensuring that the required pre-scripts have been executed as well: perl awstats.pl -update-config=vpx You can view the new statistics in a browser via the following URL:

12 7.4 Script configuration To use the scripts, replace the parameters marked with $parameter as described in Section 8 below. After the primary setup, you can add as many virtual portal instances as you want, by creating AWStats configuration for a new instance and adding the instance name (url context) to the VPS list in variables.sh. NOTE: If you modified the scripts in a Windows editor, execute the command dos2unix on the Linux/UNIX shell to remove Windows characters and convert the files to a Linux/UNIX format. 7.5 Cronjob configuration To run the scripts automatically, add the following lines to your cron configuration: # # Extensions for AWSTATS # # Update awstats database every day at 01: * * * $script root/awstats update sh 8 The script package In this section the source code of each script is listed and briefly explained. The parts that must be adapted for your environment are also marked. 8.1 variable.sh Table 9 explains the parameters of the variable.sh script. Table 9. Default values and descriptions of the variable.sh script Parameter Description Default Value date mk rm java Full path to system tool 'date' Full path to system tool 'mkdir' /usr/bin/date (UNIX) /bin/date (Linux) /usr/bin/mkdir (UNIX) /bin/mkdir (Linux) Full path to system tool 'rm' /bin/rm (Unix & Linux) Full path to java installation /usr/java14/jre/bin/java (UNIX) /usr/bin/java (Linux) perl Full path to perl installation /usr/bin/perl (UNIX & Linux) tar Full path to GNU Tar /usr/bin/tar (UNIX) /bin/tar (Linux) merger Full path to the AWStats log merger script /usr/local/awstats/tools/logresolvemerge.pl parsesa Full path to the perl script /usr/local/scripts/awstats/parsesalog.pl parsesalog.pl awstats Full path to AWStats /usr/local/awstats/wwwroot/cgibin/awstats.pl

13 awstats_script_root Installation directory of the scripts preparing the SA logs for AWStats (except: awstats_update.sh) /usr/local/scripts/awstats WPBIN The binary directory of the /usr/ibm/websphere/portal/bin local WebSphere Portal installation directory CP System library path /usr/lib log_base_dir SALOGDIR CURRDATE WPADMIN WPADMINPWD WPURL INXML XSL NEWSA VPS Log and working directory of the scripts Directory where the original site analyzer logs are located Holds information about the current date WebSphere Portal Administrator WebSphere Portal Administrator Password WebSphere Portal URL for the xmlaccess configuration interface Full path to the XML file for issuing the xmlaccess page export Full path to the transformation XSL Full path to the merged site analyzer log file List of the virtual portal context URIs separated by spaces (for the main portal enter 'root') /usr/local/awstats/salogs /usr/local/awstats/salogs `${date +%d%m%y_%h%m` Wpsadmin Passw0rd ${awstats_script_root/exprequest.xml ${awstats_script_root/transform.xsl /usr/local/awstats/salogs/samerge.log root The source code is as follows: NOTE: Blue-colored elements in this script are variable replacements for this document only; you cannot keep these during script execution. Change the values according to your environment.

14 #!/bin/bash #Setup environment variables... #Applications datefunc=/bin/date mk=/bin/mkdir rm=/bin/rm java=/usr/bin/java perl=/usr/bin/perl merger=$awstats_install_root/logresolvemerge.pl parsesa=$script_root/parsesalog.pl awstats=awstats_install_root/wwwroot/cgi-bin/awstats.pl tar=/bin/tar awstats_script_root=/usr/local/scripts/awstats #Directories WPBIN=$portal_server_root/bin CP=/usr/lib #Please do not use any variable replacements here, since these values are required for perl as well #Directory of new SA logs log_base_dir=$awstats_install_root/salogs #Directory of current SA logs SALOGDIR=$awstats_install_root/salogs #Other variables CURRDATE=`${datefunc +%d%m%y_%h%m` WPADMIN=$portal_admin_id #Portal Administrator WPADMINPWD=$portal_admin_pw #Portal Admin Password WPURL= #Websphere Portal configuration XML INXML=${awstats_script_root/exprequest.xml #XML Request file and path XSL=${awstats_script_root/transform.xsl #XSL Transformation filename and path logfile=/usr/local/awstats/log/log_${currdate #Path to and filename of merged SA log NEWSA=$awstats_install_root/salogs/samerge.log variable replacements here #Do not use any #Names of virtual Portals separated by a space character VPS="root $VPx" #entries must be entered case-sensitive 8.2 awstats_update.sh NOTE: Be careful when adapting the variable in green to your environment in the script:

15 #!/bin/bash #set -x ####################################################################### # # This script initiates the pre-awstats scripts and the update process # of AWStats # # Name : awstats_update # # PreReq : The appropriate configuration on awstats has to be prepared # # This productivity aid is provided on an "AS-IS" basis # WITHOUT ANY WARRANTY OF ANY KIND # ####################################################################### # awstats_bin=$script_root/awstats datefunc=/bin/date currdate=`${datefunc +%F_%H%m` logfile=$awstats_install_root/log/cronlog_${currdate #creating log header echo " " >> ${logfile 2>> ${logfile echo "Execution date: `${datefunc +%F`" >> ${logfile 2>> ${logfile echo "Execution time: `${datefunc +%T`" >> ${logfile 2>> ${logfile echo "Used config: $1" >> ${logfile 2>> ${logfile echo "" >> ${logfile 2>> ${logfile sh ${awstats_bin/prepare_parsing.sh echo "Process ended at `date +%T` on `date +%F` " >> ${logfile 2>> ${logfile echo " " >> ${logfile 2>> ${logfile 8.3 prepare_parsing.sh

16 Adapt the second line in the script according to your environment (in green text): #!/bin/bash source $script_root/awstats/variables.sh echo "Export site structure from ${WPURL..." >> ${logfile 2>> ${logfile for VP in ${VPS do echo "Connecting to ${VP..." >> ${logfile 2>> ${logfile LOGDIR=${log_base_dir/${VP OUT=${LOGDIR/${VP.xml CURRDATE=`${datefunc +%F_%H%M` if [! -e ${LOGDIR ]; then ${mk -p ${LOGDIR fi ${rm -f ${OUT if [ ${VP = "root" ]; then ${WPBIN/xmlaccess.sh -user ${WPADMIN -password ${WPADMINPWD -url ${WPURL -in ${INXML -out ${OUT~ else ${WPBIN/xmlaccess.sh -user ${WPADMIN -password ${WPADMINPWD -url ${WPURL/${VP -in ${INXML -out ${OUT~ fi ${java -cp ${CP org.apache.xalan.xslt.process -IN ${OUT~ -XSL ${XSL -OUT ${OUT ${rm -f ${OUT~ done echo "Merging site SA log files..." >> ${logfile 2>> ${logfile ${perl ${merger ${SALOGDIR/sa_WebSphere_Portal_*-*.log > ${NEWSA echo "Deleting original files..." >> ${logfile 2>> ${logfile ${tar -cf ${SALOGDIR/logs_${CURRDATE.tar ${SALOGDIR/sa_WebSphere_Portal_*-*.log ${rm -f ${SALOGDIR/sa_WebSphere_Portal_*-*.log echo "Start AWStats pre-process..." >> ${logfile 2>> ${logfile ${perl ${parsesa echo "Run AWStats configuration..." >> ${logfile 2>> ${logfile for VP in ${VPS do LOGDIR=${log_base_dir/${VP OUT=${LOGDIR/${VP.log echo "Updating awstats config for ${VP..." >> ${logfile 2>> ${logfile ${awstats -update -config=${vp -LogFile=${OUT echo "Deleting log for ${VP" >> ${logfile 2>> ${logfile ${rm -f ${LOGDIR done echo "GENERATE_VP FINISHED ${CURRDATE" >> ${logfile 2>> ${logfile

17 8.4 parsesalog.pl Adapt the line in green to your environment: #!/usr/bin/perl use Digest::MD5 qw(md5 md5_hex); use XML::XPath; # Declare the subroutines sub trim($); sub ltrim($); sub rtrim($); $anonymous = md5_hex("anonymous"); $variables = "$script_root/awstats/variables.sh"; print STDOUT "Opening properties file ". $variables. "...\n"; open (PROP, "<$variables") or die "an error occurred: = <PROP>; close(prop); foreach (@properties) { $propline = trim($_); if ( $propline =~ /^VPS="(.*)".*$/ ) { $vps = trim($1); if ( $propline =~ /^NEWSA=(.*.log)\s.*/i ) { $filename = $1; if ( $propline =~ /^log_base_dir=(.*)$/i ) { $logdir = $1; $xmldir = $1; print STDOUT "Properties read: \n"; print STDOUT "Logdir: ". $logdir. "\n"; print STDOUT "XMLdir: ". $xmldir. "\n"; print STDOUT "Logfile: ". $filename. = split(/ /, $vps); foreach $portal (@portals) { print STDOUT "VP: ". $portal. "\n"; #Create Hash / Mappings new sa-log-name => Input Site XML print STDOUT "Creating list of virtual portals...\n"; foreach $portal (@portals) { $xmlvpmap{$portal = $portal. ".xml"; print STDOUT "Opening sa log ". $filename. "...\n"; open (SALOG, "<$filename") or die "an error occured: = <SALOG>; # Read it into an array close(salog); # Close the file #Parse the SA Log for each XML File print STDOUT "Start processing each portal...\n"; foreach $key (keys %xmlvpmap) { $counter = 0; print STDOUT "Processing virtual portal ". $key. "...\n"; $logfile = $logdir. "/". $key. "/". $key. ".log"; $xmlfile = $xmldir. "/". $key. "/". $xmlvpmap{$key;

18 $xp = XML::XPath->new(filename => $xmlfile) or die ("Cannot open". $xmlfile); open (FILEHANDLE, ">$logfile") or die ("Cannot open". $key); #Preparse XML file print STDOUT "Parsing xmlfile ". $xmlfile. "...\n"; $nodeset = $xp->find('/export/page'); foreach $node ($nodeset->get_nodelist) { $objectid = $node->find('@objectid'); $titles{$objectid = $node->find('@title'); $unids{$objectid = $node->find('@uniquename'); \n"; print STDOUT "Parsing log file...\n"; #Go throug sa.log foreach (@lines) { $counter++; if ( $counter % 100 eq 0 ) { print STDOUT "Parsed ". $counter. " lines so far! $line = trim($_); #Only GET Page Requests are required if ( $line =~ /"GET \/Page\//i ) { #Extract username from line $line =~ /.*-\s(.*)\s\[/i; $username = $1; #Value for logname/username is not empty or '-' if ( $1 eq '-' ) { #Replace empty entry with md5 hash for anonymous $line =~ s/\s-\s-/ - $anonymous/g; elsif ( $1 eq '' ) { #Replace empty entry with md5 hash for anonymous $line =~ s/\s-\s-/ - $anonymous/g; else { #Replace clear text username with a md5 hash $username_md5 = md5_hex($username); $line =~ s/$username/$username_md5/g; #Extract objectid $line =~ /\[ObjectIDImpl_'(.*)'/i; $oid = $1; #Check for OID in XML if ( defined($titles{$oid) ) { $title = $titles{$oid; $uniquename = $unids{$oid; #Replace spaces in title & uniquename $title =~ s/\s/_/g; $uniquename =~s/\s/_/g; $oidreplacement = '"("'. $uniquename. '")"'.$title; $line =~ s/(\"get \/Page\/)(.*)($oid)(.*)(\sHTTP.*)/\1$oidreplacement\5/g; print FILEHANDLE $line. "\n"; #Clear hashes containing OID and title mappings

19 undef %titles; undef %unids; close (FILEHANDLE); # Perl trim function to remove whitespace from the start and end of the string sub trim($) { my $string = shift; $string =~ s/^\s+//; $string =~ s/\s+$//; return $string; # Left trim function to remove leading whitespace sub ltrim($) { my $string = shift; $string =~ s/^\s+//; return $string; # Right trim function to remove trailing whitespace sub rtrim($) { my $string = shift; $string =~ s/\s+$//; return $string; 8.5 exprequest.xml <?xml version="1.0" encoding="utf-8"?> <request xmlns:xsi=" xsi:nonamespaceschemalocation="portalconfig_1.4.xsd" type="export"> <portal action="locate"> <content-node action="export" uniquename="wps.content.root" export-descendants="true"/> </portal> </request>

20 8.6 transform.xsl <?xml version='1.0'?> <xsl:stylesheet xmlns:xsl=" version="1.0"> <xsl:output method="text" /> <xsl:template match="/"> <xsl:text><?xml version='1.0'?> </xsl:text> <xsl:text><export> </xsl:text> <xsl:apply-templates select="request/portal/content-node" /> <xsl:text> </export> </xsl:text> </xsl:template> <xsl:template match="request/portal/content-node"> <xsl:text> <page </xsl:text> <xsl:value-of <xsl:text> uniquename="</xsl:text><xsl:value-of <xsl:text>" objectid="</xsl:text> <xsl:value-of <xsl:text>" title="</xsl:text> <xsl:value-of <xsl:text>" /></xsl:text> </xsl:template> </xsl:stylesheet> 8.7 AWStats sample configuration (truncated) This is merely an excerpt of a configuration file. Only non-standard parameters are listed. LogFile="/usr/local/awstats/salogs/VPx/VPx.log" LogType=W LogFormat = "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot" LogSeparator=" " SiteDomain="VPx" HostAliases="VPx localhost" DNSLookup=2 DirData="/var/lib/awstats" DirCgi="/awstats" DirIcons="/awstatsicons" AllowToUpdateStatsFromBrowser=0 AllowFullYearView=2 9 Configuration alternatives Let s now discuss some alternative configurations. 9.1 Non-Portal-Server hosting AWStats In the example environment, both the HTTP server hosting AWStats and the Portal server instance are installed on the same physical server. If, in your infrastructure, Portal and Web server are hosted on different servers, you must modify the scripts a bit, since the prepare_parsing.sh script requires xmlaccess to export the page hierarchies.

21 In this case, copy the scripts to the server hosting WebSphere Portal and comment the following lines in prepare_parsing.sh: echo "Run AWStats configuration..." >> ${logfile 2>> ${logfile for VP in ${VPS do LOGDIR=${log_base_dir/${VP OUT=${LOGDIR/${VP.log echo "Updating awstats config for ${VP..." >> ${logfile 2>> ${logfile ${awstats -update -config=${vp -LogFile=${OUT echo "Deleting log for ${VP" >> ${logfile 2>> ${logfile ${rm -f ${LOGDIR echo "Now removing ${OUT..." >> ${logfile 2>> ${logfile ${rm -f ${OUT done Provide the new site analyzer logs to the AWStats server by establishing, for example, an NFS share, and then create another script on the server hosting AWStats, to execute the code above. 9.2 Anonymous usernames To display the usernames in clear text and not as md5 hash, comment the following lines in parsesalog.pl: if ( $line =~ /"GET \/Page\//i ) { #Extract username from line $line =~ /.*-\s(.*)\s\[/i; $username = $1; #Value for logname/username is not empty or '-' if ( $1 eq '-' ) { #Replace empty entry with md5 hash for anonymous $line =~ s/\s-\s-/ - $anonymous/g; elsif ( $1 eq '' ) { #Replace empty entry with md5 hash for anonymous $line =~ s/\s-\s-/ - $anonymous/g; else { #Replace clear text username with a md5 hash $username_md5 = md5_hex($username); $line =~ s/$username/$username_md5/g; 9.3 XMLAccess export The frequent xmlaccess export of the Portal site structure can be skipped if (1) you provide the XML manually or (2) no virtual portals are used 1. The following adaptations are required to switch off this part of the script: 1. Adapt parsesalog.pl as follows: with URL replacement using xmlaccess: 1 Actually the whole URL string could be shortened to display only the page title, but if virtual portals are used and a separate statistic for each portal shall be created, the xmlaccess export is required. Without the XMLs of each portal, the script cannot parse the logs correctly.

22 #Extract objectid $line =~ /\[ObjectIDImpl_'(.*)'/i; $oid = $1; #Check for OID in XML if ( defined($titles{$oid) ) { $title = $titles{$oid; $uniquename = $unids{$oid; $oidreplacement = '"(". $uniquename. ")".$title. "\"'; $line =~ s/(\"get \/Page\/)(.*)($oid)(.*)(\sHTTP.*)/\1$oidreplacement\5/g; print FILEHANDLE $line. "\n"; without xmlaccess: #Extract objectid $line =~ /(.*\]\/)(.*)(\shttp.*\")/i; $replacement = $2; $line =~ s/(\"get.*)(\/page.*)(\shttp.*)/\1$replacement\3/g; print $line. "\n" 2. Comment the following lines in parsesalog.pl : $xmlfile = $xmldir. "/". $key. "/". $xmlvpmap{$key; $xp = XML::XPath->new(filename => $xmlfile) or die ("Cannot open". $xmlfile); #Preparse XML file print STDOUT "Parsing xmlfile ". $xmlfile. "...\n"; $nodeset = $xp->find('/export/page'); foreach $node ($nodeset->get_nodelist) { $objectid = $node->find('@objectid'); $titles{$objectid = $node->find('@title'); $unids{$objectid = $node->find('@uniquename'); 3. Comment the following lines in prepare_parsing.sh: ${rm -f ${OUT if [ ${VP = "root" ]; then ${WPBIN/xmlaccess.sh -user ${WPADMIN -password ${WPADMINPWD -url ${WPURL -in ${INXML -out ${OUT~ else ${WPBIN/xmlaccess.sh -user ${WPADMIN -password ${WPADMINPWD -url ${WPURL/${VP -in ${INXML -out ${OUT~ fi ${java -cp ${CP org.apache.xalan.xslt.process -IN ${OUT~ -XSL ${XSL -OUT ${OUT ${rm -f ${OUT~

23 4. Set the value of the property VPS in variable.sh to root, to process the main portal only: VPS="root #entries must be entered case-sensitive 10 Conclusion AWStats for WebSphere Portal 6.0 is a free and powerful solution to track user activities and display the information graphically. You have learned how to easily implement an add-on to extend AWStats functionality for virtual portals and human-readable page descriptions in the resulting statistical analysis. 11 About the author Markus Moltenbrey is an IT Specialist for WPLC infrastructure and architecture within IBM's Software Services for Lotus. He began his career with IBM four years ago, working on Lotus Workplace and WebSphere Portal customer projects. The author can be reached at moltenbrey@de.ibm.com. 12 Resources AWStats installation instructions: developerworks WebSphere Developer Technical Journal: Using portal analytics with opensource reporting tools: WebSphere Portal Information Center: Apache HTTP Server: CPAN: IBM JRE Download page: Trademarks IBM, Lotus, Tivoli, and WebSphere are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others.