Instant Message Retrieval GoGetIM. Michael Seltzer Faculty Advisor: Zachary Ives

Transcription

1 Abstract Istat Message Retrieval GoGetIM Michael Seltzer Faculty Advisor: Zachary Ives May compaies ad idividuals have recetly developed a wide variety of software that attempts to moitor, store, ad orgaize the vast collectio of iformatio that we iteract with as computer users. Applicatios such as Google Desktop Search ad MSN Search Toolbar allow users to easily fid ad access a wide variety of data, from to word processig documets. However, most of these applicatios igore the characteristics ad itricacies that ca be associated with alterative types of iformatio. Istat messages are a prime example of iformatio that is fudametally differet from other web or local iformatio. They are essetially threads of back-ad-forth messages occurrig i spurts; however, they are curretly ot represeted i this maer. Cosiderig these complexities, traditioal idexig ad retrieval procedures caot practically apply. Savig the coversatio iformatio is relatively easy; however, idexig, orgaizig ad user retrieval i a practical fashio is the ultimate challege. The goal of my system was therefore to improve retrieval methods for coversatios. While I primarily wated to improve the retrieval process of searchig for istat message coversatios, a subset of fuctioalities were uderlyig this goal. This icluded searchig by simple message cotet or by a advaced search that specified other costraits. A example of a more advaced search capability would be for a user to search for a message from coversatios with a specific subset of users, ad withi a certai rage of dates. This goal also icluded producig search results that reflect the specific coversatio the user is seekig, rather tha the etire log file of the user s coversatios of the day or the sigle lie i this log that cotais it. Thus, logs ca be shaped ad preseted i their origial coversatioal cotext, istead of appearig withi the cocateated ad cumbersome file. - 1 of 13 -

2 Related Work There is curretly a diverse selectio of commercial applicatios that archive ad retrieve both web-based ad local iformatio. Search egies such as Google ad Yahoo! have developed ad released desktop search software with the capability of archivig ad retrievig istat message coversatios. However, curret search techology for istat messages is mostly limited to simple text-matchig. Additioally, some istat messagig cliets or software add-is curretly store coversatio history; however, retrieval methods are restricted to basic search across all log files ad the uisace of maual discovery. For example, DeadAIM archives istat message coversatios; however, its built-i search is rudimetary ad ieffective. Hece, there is room for improvemet i establishig efficiet techiques for accessig istat message coversatios. The system that I implemeted far surpasses these curret systems i terms of the user s ability to specify search limitatios as well as the relevace of iformatio that the program retrieves. Techical Approach I order to maitai the focus of my project o the retrieval of istat message coversatios, I did ot build ay compoets to archive coversatios. Istead, I used the output of DeadAIM software for archived istat message data. As oe of may differet third-party plug-is for AOL Istat Messeger (AIM), DeadAIM automatically archives istat message coversatios. DeadAIM produces a simple HTML file that cotais all of the coversatios oe has with user X o date Y, i chroological sequece. Because I already had a collectio of these files accessible due to my previous use of the software, the approach of usig these files was by far the most ideal. I additio to usig DeadAIM, I exteded Apache Lucee, a ope source java program that ca be molded to idex ad search a variety of data. I used Lucee to help build a istat message coversatio idex ad provide search fuctioality for that idex. O the top of the applicatio is a GUI that I built usig Java s Swig compoets that allows the user to iteract with the fuctioig code that I have writte. The fuctioal code is separated ito two geeral compoets: a idexig compoet ad a searchig compoet. The idexig compoet parses through all DeadAIM HTML logs to create a idex of coversatios, ad the - 2 of 13 -

3 searchig compoet provides results from a created idex relevat to multiple user-supplied costraits. Figure 1: Overall Techical Implemetatio Diagram Graphical User Iterface Idexig Compoet Searchig Compoet DeadAIM Archives Coversatios Userames Date Rage Search Query Coversatio Idex Search Results Idexig I created a package of classes called imstrip that would cotai all of the code pertaiig to idexig istat message coversatios. Iside this package resides the Strip class, which hadles all parsig for a idividual log file. Each log file cotais all istat messages set to ad received from a particular user o a particular date. Sice AIM produces horribly formatted HTML for these coversatios, each Strip istace is passed a versio of this HTML cleaed up - 3 of 13 -

4 usig the ope source program JTidy. This cleaed HTML is traversed, ad all message userames, times ad message texts are collected. The IMStrip class, also withi the imstrip package, fids all relevat log files for a Strip istace to parse. Oce these HTML log files have bee parsed out ito userames, times ad message texts, the IMStrip istace uses the Lucee data class Documet to orgaize all of the iformatio ito coversatios. While each Documet ormally cotais fields for files such as title, date, locatio ad so o, I implemeted each Documet to have fields for the remote userame, the local userame, the date of coversatio, ad fields for every message set. The message text, or field value, is give based o the regular expressio: (r l)(m a)(######)(.+). The r is for a message that the remote user set, or the l would be for a message that the local user set. The m is for a ormal message, or the a would be for a automatically set message (through a AIM away message fuctio). The (######) represets the time of the message i 24-hour format. The actual message text follows. Thus, whe a remote user seds a regular message at 2:54:06 PM, sayig hi there the idexer would create a message field with the value rm145406hi there. After much cosideratio, I decided that time would be the best idicatig factor for the begiig ad ed of coversatios. As a mai focus of this project was to be able to supply relevat coversatioal iformatio to the user, it was importat to have a cosistet ad appropriate way to determie what the actual coversatios were. Most of the time olie coversatios cosist of back-ad-forth messagig. Whe there are gaps of time i betwee messages, such a lapse either idicates a shift to a ew coversatio, or that a user has temporarily switched his focus away from the coversatio ad to aother task. The former case aloe was sigificat eough of a idicator for coversatio boudaries to outweigh the istaces of the latter. Therefore, give a variable of a legth of time, coversatio boudaries were established wheever the time betwee messages was larger tha this variable. Oce all coversatios are established iside the IMStrip istace, they are writte to a idex via the Lucee IdexWriter class. I also write a list to the idex of userames as well as - 4 of 13 -

5 begiig ad edig dates that the idex cotais for easy access for the GUI. This idex is stored i the give locatio for the search ed of the program to access. Idexig Challeges The idexig compoet was by far the more challegig ed of the project. It would be a uderstatemet to describe AIM-geerated HTML as ill-formatted ad sloppy. Eve with help from JTidy to clea it up, I ecoutered sigificat problems with parsig. For example, there was a large icosistecy with regards to how the begiigs of messages were represeted. Whe JTidy created a DOM tree of HTML compoets for each log file, geerally the body ode for each file cotaied a spa child ode that further cotaied all of the iformatio relevat to oe message set or received. However, for some idistiguishable reaso, some spa childre would be located as childre of other spa odes istead of the origial body ode. These odd spa odes would the be wrogly recogized as message text for these wrogly established paret spa odes. I eeded to perform some sigificat workarouds i order to restore fuctioality ad address this problem. I more geeral terms, it was difficult to come up with a simple ad comprehesive algorithm to parse through the dese ad cofusig data that is AIM HTML logs. Each time I believed that my method parsed through files correctly, I would ecouter files that represeted similar data i a differet fashio that coflicted with my method. My example metioed above was but oe istace of similar data represeted i multiple, icosistet forms. Maagig all of these differet icosistecies, special cases, ad variatios at the same time proved to be a very difficult task. The method I have put i place should properly parse 99.9% of the files that AIM produces. Although I am somewhat disappoited that my project is ot bug-free or completely comprehesive, give the circumstaces ad the still extremely high parsig yield, the result is certaily more tha satisfactory. A importat idexig issue that I ecoutered that did ot pertai to bad HTML data was related to how ad whe DeadAIM writes AIM logs. DeadAIM writes all istat message coversatio to log files as soo as the user closes his istat message widow. However, if the widow cotais istat messages that spa across multiple days, these messages are still all - 5 of 13 -

6 writte together i sequece i the same file. For example, say user X has a coversatio with user Y that spas from 11:55 PM to 12:05 AM of the ext morig, at which poit user X closes the widow. All messages, both from the previous day as well the ext, are writte i the same HTML file ad are filed uder the later day. What is more, there is o record of what the previous date(s) is. Thus, a user could keep a istat message widow ope for days, have multiple coversatios o multiple days with a user, ad all messages would be writte to the log as if they occurred o the day the log file was writte. Sice there was o way for me to detect the dates for messages ot set or received o the log file dates, I dealt with the issue i the maer I believed to be most appropriate. I detected ay message sequeces where the time of a message was before the time of the previous message set. This would idicate a message set or received o a ew day. I was forced to assume that the earlier message was set or received o the previous caledar day, as there was o other iformatio to referece. These essetially udated coversatios, while give a assumed date i my idexig compoet, are always starred for the program user to iform him that the date associated with the coversatio may or may ot be appropriate. Searchig Searchig withi a idex built with the idexig compoet of my program makes use of Lucee as well. Through the GUI, a user ca specify a subset of userames, a subset of dates, ad a search query. My program the builds a query for Lucee to search usig these three factors. The set of userames selected is matched with the remote userame field s value stored withi each coversatio. Similarly, the set of dates selected is matched with the date field s value stored withi each coversatio. The search query is matched with all message text values. A istace of the Lucee class Hits is retured by the Lucee search fuctio, which cotai all of the relevat Documet istaces, or essetially all of the coversatios that match the search criteria. Lucee uses a form of tf-idf weightig i order to score or rak the set of results that are retured. This score is o the scale of 0 to 1. I tweaked this score i order to better reflect the rakig of coversatio results. This ew coversatio score is still o the same scale, ad is - 6 of 13 -

7 wi i calculated as L *( ) *(( ) / dˆ ) where L is the Lucee score, is the umber of messages i ˆ the coversatio, ad ˆ is the maximum umber of messages i a coversatio across the etire idex. Thus, ( ) is a weightig of the relative legth of the coversatio. Beyod this, w i is the ˆ wi i umber of words i each message i i the coversatio, which meas that ( ) is equivalet to average message desity for the coversatio. The term dˆ is the maximum average message wi i desity i a coversatio across the etire idex. Thus, (( ) / d ˆ ) is a weightig of the relative average message desity of the coversatio. I all, the ew score that I compute is the product of the Lucee score, the relative legth ad the relative average message desity. This ew coversatio score allows for a better rakig of coversatios. Sice these results are oly retured sorted by their Lucee scores, I reorgaize them accordig to my ewly calculated coversatio score. These sorted results are preseted i successio with the search terms highlighted. I additio, I provide a iterface for the program user to traverse the search results for a particular user, ad subsequetly for a particular date. For example, if the program user is preseted with a set of results ad realizes that coversatios with a particular userame were really what he was lookig for, he ca easily avigate to those results for that userame without havig to perform a ew search with modified parameters. Whe the program user decides to traverse these results subsets, the subsets are agai sorted by coversatio score. The implemetatio of this part of the iterface is further described i the sectio pertaiig to the GUI that I developed. Searchig Challeges The searchig compoet of my program was oe of the less challegig aspects of the project. As Lucee already provided a very comprehesive ad efficiet search fuctio with which to - 7 of 13 -

8 access the database, most of my implemetatio focused o dealig with the results obtaied from the Lucee search. At first I was cocered that for each additioal costrait provided, such as each additioal userame to search, I would have to perform a separate query. I implemeted this method ad foud that it was very resource itesive to search across all userames ad the etire date rage. Sice I imagied this would be a very commo iteractio with the program, I sought out a more efficiet search implemetatio. Usig Lucee s MultiFieldQuery class, I was able to costruct a sigle query for multiple field types (i.e. userame, date, message), cocateatig the costrait subsets with the OR operator. This allowed my program to make full use of Lucee s efficiet backboe, ad it vastly improved the resource requiremets ad performace of my program. Graphical User Iterface With the help of NetBeas IDE s Matisse GUI builder, I was able to build a robust GUI for my program that provides as a iterface for both the idexig ad searchig compoets. The program user, upo loadig my program, first chooses where his AIM log files are located. The program user the chooses where to save the idex that the program will compile. Fially, the user specifies via a slider bar the legth i secods to serve as a cutoff for coversatio boudaries. After the idexig begis, the GUI periodically is updated with the status, as the percet complete is displayed alog with the userame for whom logs are curretly beig idexed, which cotiues util idexig is complete. The GUI also allows the user to locate a previously created idex to load. I either case, a full idex is the loaded by the program. - 8 of 13 -

9 Figure 2: Idexig Logs The search iterface sectio of the GUI allows the user to specify the costraits of a set of userames ad dates as well as a search query. The list of userames is displayed withi a JList to allow for multiple selectios. The rage of dates is specified usig a start date ad a ed date, selectable via a series of dropdow meus. I costructed a variety of easy-access buttos to help the user select commo userame sets ad commo date rages, such as Select All userames ad All Dates. Beeath the search query, the user has the optio of eablig the Expad Terms fuctioality. Give the ature of coversatio text, there ted to be a wide variety of abbreviatios, acroyms ad slag. For example, istat message users will ofte make use of terms such as brb for be right back ad ttyl for talk to you later. With the Expad Terms fuctio eabled, these shorthad message compoets are expaded to their full text equivalets. After specifyig users ad dates, ad eablig or disablig the expasio of shorthad terms, the user eters a search query ad the GUI refereces the search compoet to gather results. - 9 of 13 -

10 The coversatio results, as metioed earlier, are first displayed i oe large pael, sorted by coversatio score. Located o the side of this pael is a series of tabs, with each tab labeled with a userame that cotais coversatio results. If the user selects oe of these tabs, the pael that origially displayed all coversatio results chages to cotai all coversatio results with that userame. Additioally, this pael also cotais a ew series of tabs, located o the top, with each tab labeled with a date that cotais coversatio results for that userame. If the user selects oe of these tabs, the results pael chages oce agai to cotai all coversatio results with that userame o that particular date. To illustrate, visualize a certai search s results that retured coversatios with users X ad Y from both today ad yesterday. The program would at first show all of these coversatios. The user could select a tab labeled X ad the program would the show the coversatios with X from both today ad yesterday that are i the search s results. The user could the select a tab labeled with today s date ad the program would the show the coversatios with X from today that are i the search s results. This provides for extremely easy avigatio of results from multiple agles, thus removig the eed to redo search i may users cases. This search fuctio ad subsequet avigability is a vast improvemet over curret software s abilities whe it comes to istat message retrieval of 13 -

11 Figure 3: Searchig ad Navigatig Results Graphical User Iterface Challeges While the Matisse GUI builder provided a easy iterface for me to create the basic GUI for my program, creatig the results pael was more challegig. The Matisse iterface does ot allow for dyamic compoets that are created at rutime. Rather, this must all be created with separate code. I foud it to be extremely tricky to create all of the user tab paels ad date tab paels whe search results were cojured. Oce I fully uderstood all of the GUI Swig code that I eeded i order to create each idividual compoet, I was able to devise a algorithm that would build this iterface. I made extesive use of vectors of Swig compoets i order to keep track of what compoets I had created as well as what further Swig compoets would eed to be associated with them. Whe a ew search is performed, all previous Swig compoets from the previous search results are wiped out ad recreated accordig to the ew results of 13 -

12 The GUI buildig part of my project was oe of the more iterestig processes for me. I had little previous experiece workig with GUIs, ad it was very satisfyig for me to have a visually appealig iterface that I created correspod with the backboe code that I had previously produced. I leared a great deal with regards to the fuctioality of Java s Swig compoets ad how they ca be used. I the ed, I was much more comfortable with my ability to geerate a GUI for ay fuctioality that my program required. Coclusios ad Reflectios Whe I preseted my program to my frieds ad classmates who had all previously used DeadAIM to access their istat message logs, they were all extremely impressed with the fuctioality ad possibilities for the program. I pla o submittig my program as ope source software to a variety of olie commuities whose users could make use of it. I o doubt see my program as a vast improvemet over previous istat message retrieval systems. Cosiderig my accomplishmets, I believe that my program meets my overall goal of expadig the possibilities of istat message retrieval. The process of buildig the program provided a wide array of challeges for me to overcome. I was disappoited i the amout of time that I evetually eeded to dedicate to steady the istat message parsig code. I was eve more disappoited to fid towards the ed of the project that a very recetly released versio of AIM software logs all istat message coversatios i XML format, which would be easy to maipulate i ay program. I ca oly imagie what other aspects of this program I could have focused o give this superior iformatio from the oset. Regardless, I am cotet with my experiece of fightig the beast that was the iferior data. Most importatly, I was very impressed with the ed product s usability. To complete this project ad be able to have somethig tagible ad valuable was very satisfyig. I am overwhelmed with a sese of pride whe I show off my program s features. I ca say that my program efficietly parses ad idexes vast collectios of istat message logs ad the provides a easy ad comprehesive iterface to retrieve them from the resultig idex. That aloe is a lot more tha what I could have said from this project s begiig of 13 -

13 Refereces 1. Google Desktop Search, 2. MSN Search Toolbar, 3. DeadAIM, 4. Apache Lucee, 5. JTidy, 6. NetBeas IDE, of 13 -