cmd p p p cmd cmd ... cmd p p p command 1 command 2 command 3 cmd cmd ... cmd S M T W T F S command n cmd ... cmdn p p p cmd ...

Transcription

1 The Trials and Tribulations of Building an Adative User Interface Benjain Korveaker & Russell Greiner fbenjain, Deartent of Couter Science University of Alberta Edonton, Canada Abstract As every user has his own ideosyncracies and references, an interface that is honed for one user ay be robleatic for another. To accoodate a diverse range of users, any couter alications therefore include an interface that can be custoized e.g., by adjusting araeters, or dening acros. This allows each user to have his \own" version of the interface, honed to his secic references. However, ost such interfaces require the user to erfor this custoization by hand a tedious rocess that requires the user to be aware of his ersonal references. We are therefore exloring adative interfaces, that can autonoously deterine the user's reference, and adjust the interface aroriately. This aer reorts a series of exerients towards building such an adative interface here a Unixshell that can redict the user's next coand based on his revious interactions, and use this to silify the user's future interactions. After suarizing the Davison/Hirsh (998) work (for learning \coand stubs"), we then exlore several ways of extending and iroving this syste; e.g., to redict entire coand lines, to use various other tyes of inforation, etc. Keywords: online learning, learning and adatation, learning user references Introduction There are today a wide variety of interactive couter alications, ranging fro web-browser and searchers, through sreadsheets and data-base anageent systes, to editors, as well as gaes. As these systes becoe ore colicated as required to be able to accolish ore tasks, better their interfaces necessarily also becoe ore colex. Many of these systes have begun including tricks to hel the users; e.g., if the user begins an ety le with \Dear John", Word will suggest a \Letter" telate; siilarly if the user begins a line with \*", Word will change that character to a bullet \" and go into its List environent. Unfortunately, dierent users have dierent references, which eans the tricks that are aroriate for one user ay be robleatic for another. (E.g., there aarently are eole who like the Microsoft \Oce Assistant"... ) Moreover, dierent users want to do dierent things with the syste, as they have very dierent abilities, background knowledge, styles, etc. This realization that \one size does NOT t all" argues for custoizable interfaces, that can rovide dierent interfaces for dierent users, and hence allow each user to have an interface that is honed to his individual references. Of course, any of today's alication rogras can be custoized; e.g., ost editors and shells include acroor scriting- facilities. However, this custoization rocess ust tyically be done by the user which eans, tyically, that it is not done by the user, as this custoization rocess () requires that the user knows how to ake this odication (e.g., knows both the naes of the relevant araeters, and how to odify the); () requires the user to be aware of his secic references, and (3) is usually quite tedious. This research roject, therefore, ushes on a dierent aroach: Build alication systes that can autonoously adat theselves to the individual users. In articular, we focus on techniques for detecting atterns in the user's interactions, and then using this inforation to ake the interaction siler for the user, erhas by autoatically re-setting soe syste araeters, or dening aroriate new acros. This aer investigates a secic anisfestation of this task: We have built a Unix coand shell that can redict the user's behavior fro his revious coands, and then use these redictions to silify his future interactions with the shell. The rest of this introductory section resents two illustrative exales to hel describe our task ore recisely. After Section rovides the background for this work, Section 3 then sketches our algorith, based on the earlier Davison/Hirsh syste (DH98). Finally, Section 4 resents our eirical results, over a large existing dataset of 86 users. It discusses in articular the range of studies we ran, to better understand this challenge. All told, this aer resents what has been tried before, sua-

2 % vi crossword.c % ake crossword cc -g crossword.c -o crossword -l % crossword uzzle0 bus error (core dued) % vi crossword.c % ake crossword cc -g crossword.c -o crossword -l % crossword uzzle0 Solution found in 5584 attets. % crossword uzzle0 segentation violation (core dued) % ddd crossword core % vi crossword.c % ake crossword cc -g crossword.c -o crossword -l Undefined sybol: rint first referenced in file crossword.o ld: fatal: Sybol referencing errors. No outut written to crossword *** Error code ake: Fatal error: Coand failed for target `crossword' % vi crossword.c % ake cc -g crossword.c -o crossword -l % crossword uzzle03 Solution found in 3 attets. % el -s"it works" fred Figure : Wila's Coand Sequences rizes what we have learned, and suggests soe of the robles. Exales: Figure shows Wila's interactions with a shell, as she works on her crossword-solving rogra. It is easy to see that there is a consistent attern within Wila's activities; e.g., the vi-ake-crossword sequence is reeated several ties. Although Wila has taken advantage of traditional Unix facilities, she still has to tye a nuber of characters. Now exaine Fred's coand sequences; Figure. He has already coleted the assignent and is trying to write u his results. Unfortunately, Fred is not very failiar with L A TEX and is having trouble foratting an equation. Here, an even ore obvious attern is visible. Note that, if Fred had had a scrit to erfor the latexdvis-ghostview stes for hi, he wouldn't have ade the istake of running dvis on a L A TEX source le. Task: Our iediate goal is a Unix shell that can anticiate the user's next coand, and use this inforation to silify his interactions. One way to use that inforation would be to ll the user's current buer with this redicted coand e.g., after \vi % vi crossword.tex % latex crossword.tex % dvis crossword.dvi % ghostview crossword.s % vi crossword.tex % latex crossword.tex % dvis crossword.tex % ghostview crossword.s % vi crossword.tex % latex crossword.tex % dvis crossword.tex dvis:! Bad DVI file: first byte not reable % dvis crossword.dvi % ghostview crossword.s % wall Can anyone hel e with latex??? ^D Figure : Fred's Coand Sequences crossword.c", this shell could load \ake crossword" into Wila's buer. Wila could then execute this coand by sily tying a carriage return. Alteratively, she could delete this buer, or a ortion thereof, and retye whatever else she wishes. Siilarly, after Fred's \latex crossword.tex", the shell could suggest \dvis crossword.dvi"; assuing Fred acceted this suggestion, this could save Fred the hassle of guring out why \dvis crossword.tex" did not work. However, as this single ost-likely coand is still very unlikely, we decided to use the slightly dierent aroach of suggesting the 5 ost likely coands. As shown in Figure 3, our shell will dislay (in the to of the screen) these 5 ost likely coands. It also binds the to the F through F 5 keys; the user can execute any of these coans by sily ressing the associated function key. Notice either otion eans the user can achieve his results with less tying, and so do his work in less tie, while aking fewer istakes. In a nutshell, our shell will use this record of revious coands to redict what coand the user is likely to use next. These exales illustrate why our aroach has a chance of succeeding: Peole tend to follow atterns. For exale, after a successful latex coand, any users then tye dvis to roduce a ostscrit le fro the dvi le that latex generated; and if that succeeds, use ghostview to dislay the generated aer. Later versions of this tye of shell could take ore sohisticated actions based on this inforation. For exale, they could dene (and infor the user of) Note that even coands that see rando, such as \df", \utie" or \readail", are robably (robabilistically) redictable based on soe external event. Of course, this ay require elaborate instruentation.

3 Figure 3: Dislay of the Ileentation new acros, corresonding to (variabilized fors of the) sequences observed ost often. Or the adative shell could re-fetch inforation for the next anticiated coand(s) e.g., get the fonts for the ossible \dvis" coand, swa out rogras to reetively ake roo for netscae, swa out netscae before starting soe color-intensive task, or forat the ucoing an age. They ight also be able to detect atterns that corresond to robles e.g., that the user is thrashing and then rovide useful assistance. Background The extended version of this aer rovides a corehensive survey of adative user interfaces (e.g., (Lan97; SH93; HBH + 98) and others) and other work related to our task. Here, for sace reasons, we can only discuss the two ost relevant revious results. Our roject is a direct extension of the seinal work by Davison and Hirsh, (DH97; DH98) which redicts user coand stubs (coands without otions and araeters, e.g., the \latex" of the coand \latex foo.tex") fro the user's revious coand stubs. Their hand-crafted algorith generates and uses a table whose hi; ji entry is the robability of coand stub s i occurring iediately after the stub s j ; i.e., P ( Stub t+ = s i j Stub t = s j ) where the rando variable Stub i denotes the stub tyed as the user's i th coand. After seeing the current stub \stub t ", their erforance syste could then redict the ost likely subsequent stub stub t+ = argaxfp ( Stub t+ = s j Stub t = stub t )g s () The ain challenge, then, is how to ll this N N table of nubers (where there are N ossible stubs). The \obvious" aroach, of using eirical frequency over the sales observed, is arguably aroriate only if the data is iid (indeendent and identically distributed), which eans in articular that the frequency of seeing soe coand does not change over tie. That assution is clearly false in our situation. Instead, their algorith used a dierent way to estiate these robability values, one designed to ehasize recent new coands. On seeing say the coand \cd " followed by \latex " it would rst decay the robabilities on the \cd line" of the table by an (eirically deterined) factor < i.e., it would rst reset, for each stub s, Note here that ^P ( Stub t+ = s j Stub t = cd ) = () X s ^P ( Stub t+ = s j Stub t = cd ) = as this suation had been before this reduction. They then add the reaining? quantity to the hlatex; cdi entry: ^P ( Stub t+ = latex j Stub t = cd ) +=? (3) Hence, this row continues to su to, which is why we can view these values as robabilities.) This rule treendously boosts the value of ^P ( Stubt+ = latex j Stub t = cd ), even if its revious value was 0. This eect is extreely useful; e.g., Davison/Hirsh note that 0% of coands are sily the reeating the iediately rior coand!) We will later refer to this algorith as the \Alha-Udating Rule" or \AUR". Davison and Hirsh could then redict the single ostlikely coand, stub t+ (fro Equation ); however this was correct only 39.9% of the tie. They therefore switched to redicting the ve ost likely coands, and found that the actual stub (i.e., the one the user actually entered) was in this to-5 list alost 3/4 of the tie. Evaluation Criterion: Our basic algorith is a direct extension of theirs; see Section 3. We also adoted their evaluation criterion: Our systes are on-line: after observing the sequence of stubs, hstub ; : : : ; stub t i, each then redicts stub t+ (Equation ), or erhas the to ve: fstub t+; : : : stub 5 t+g. It is then told the correct stub t+ that the user actually tyed, which it can use rst to udate its \classier", and then to redict stub t+, (or erhas fstub t+; : : : stub 5 t+g), etc. We dene \accuracy" as how frequently the redicted coand coletely atches the user's actual coand. That is, after K instances, we coare the correct sequence hstub ; : : : ; stub K i to our rediction hstub ; : : : ; stub Ki, and reort as accuracy the nuber of ties \stub i = stub i ", divided by K. In They liited n to 5 as ore than ve coands tyically causes the user to focus on the redictions rather than the task at hand (or ignoring the redictions altogether).

4 the \to 5" case, the syste gets a \oint" whenver stub i fstub i ; : : : stub 5 i g. We use the sae criterion, utatis utandis, for our studies. Acknowledging that the rediction syste ay take a while to \lock onto" the articular erson, we soeties ignore the istakes ade during the rst, say, 00 coands. Here, the accuracy would be the nuber of ties \stub i = stub i " over tie i=0..k, divided by K? 00. (We will later refer to this as \accuracy?00".) Other eole have done work in the area as well. In articular, Greenberg (Gre88) collected a large dataset for 68 users, and used this to classify users into novice users, exerienced users, scientists and non-users (see Table below). Although these datasets were collected 0 years ago, they still are the largest and ostcolete source of user coand histories that are ublicly available. We use this dataset (albeit for different uroses) in the studies reorted here. 3 We nally note that, while the concet of coand rediction has been exlored for at least ten years, nothing has yet been deloyed into the ainstrea arket. We believe this can be artially exlained by observing: Couting resources ay not have been sucient to adequately redict user coands, until recently. It is dicult to collect sale data. Unix users are a territorial bunch, with their favorite editors, OS variants, and shells. 4 It is tyically very dicult to craft data collection echaniss that are transarent to the user. While the zsh source code is relatively organized and legible (esecially when coared to tcsh source code), it still took 30 hours to nd and change the aroriate 5 lines of code. 3 Prediction Algorith As stated above, our adative shell onitors the coands tyed by the user (along with other associated inforation, see below) then uses this inforation to redict, after each coand, which coand the user will tye next. To reduce the invasiveness of the data collection, we odied the Unix shell, zsh, 5 both to 3 To silify this resentation, we are not included the ore recent, but less colete, data obtained fro our ileentation. 4 One otential subject exlained why he turned us down: \... switching shells is like cutting o y ngers and sawing (sic) the on backwards and I have to relearn how to use y thub as y little nger... " 5 We chose zsh as it clais to be able to eulate the ore oular shells (tcsh, csh, bash, ksh), and it contains ost of the echaniss required to collect the araeters we want. collect the relevant inforation, and then to reset the dislay and the function keys, as shown in Figure 3. The real challenge, of course, is the actual rediction algorith, which deterines, for each ossible coand, the conditional robability that will be the next coand, based on the earlier coands, etc. In articular, we need to coute the robability that the next coand will be t+, based on the available inforation i.e., P ( Cd t+ = t+ j t; : : : ). Moreover, we need to roduce a good estiate to this distribution after very few sales, which is further colicated by the realization that the \local distribution" (i.e., the robability that one coand will iediately follow another) is not stationary. As we want a rediction syste that can track the user's distribution of coands, we therefore lift the Davison and Hirsh idea of ehasizing the iediately rior coand, then decaying the robabilities over tie; see the AUR (Section, Equations and 3). However, we extended their work in two signicant ways. First, we want to redict entire coand lines, rather than just coand stubs. Although redicting the next coand stub ight save a few keystrokes, Unix stubs are notoriously short in fact, over our dataset, the average stub length is only 4. characters. Assuing the redictions are aed to a function key and that the enter key ust be ressed afterwards, reducing the nuber of keystrokes fro four to two is not a signicant savings. Further, when one considers that the function keys are usually not articularly close to the \hoe keys", the eort sent reositioning the hand ay well negate the tie saved. Note, however, the average length of the entire coand line is 9.7 characters; and reducing 9.7 to would be useful. We therefore focus on this task, even though it is, of course, uch harder, esecially for coands that haen frequently, but with highly varying arguents e.g., cd, ls, vi and finger. The second extension was to hel us solve this harder task: We want to allow our redictor to use other inforation in roducing these redictions such as the entire revious coand line, the error code of that revious coand, the tie of day and day of week, etc. DataStructure to Encode Probabilities: We could iagine using a huge ulti-diensional table for this, whose hi; j; k; `; i entry is the conditional robability that P ( Cd t+ = i j Cd t = j; Error = k; tie = `; day = ), where i and j vary over the ossible coand lines, k over the range of ossible error codes, ` over the ties in a day and over the 7 days in the week. Unfortunately, our dataset included over 60,000 dierent coands lines. Even if we ake this secic to the individual users, note there was soe individual

5 coand coand coand 3 coand n n n n n 3 3 n n n n Y M A E N S M T W T F S Y M A E N S M T W T F S Y N N N error error error M A E N tie tie tie day day S M T W T F S day Figure 4: Structure Encoding Probabilities users that used over 3000 dierent coands! 6 Even if we quantize the error codes to only two values (0 or non- 0), and the tie to only hours, this table would need to have over = 3; 04; 000; 000 entries which would be dicult even to store, uch less to estiate fro a aucity of data (BD77). Recall that we want our syste to be able to hel users after observing only a sall handful of interactions. To reduce the nuber of araeters, we instead used a dierent structure, which ioses a bound on the nuber of ossible current coands (set to n), and on the nuber of ossible redicted coands (set to ). We also let the set of redicted coands vary, deending on the current coand. Here, we need only aintain the \active" subset of the coandairs h i ; j i, based on the largest P ( Cd t+ = i j Cd t = j ) values. We also further quantized the tie values, to only the 4 values: orning, afternoon, evening and night. The resulting structure is shown in Figure 4. To exlain the notation: the value of coand 7 ight be \latex crossword"; it oints to an associated j table. 7 There, erhas 7 is \dvis crossword.dvi" with P7 = 0:3; and 7 = \vi crossword.tex" with P7 = 0:09; etc. The value stored in Y 7 = P ( Cd t+ = dvis crossword.dvi j Cd t = \latex crossword.tex"; ErrorCode > 0 ) 6 All told, the Greenberg data involved over 6,774 distinct coands, of the total of 303,68 coands tyed by the 68 users. (Table rovides ore details.) Soe users used as few as 35 dierent coands, while one user used 353. The corresonding values for stubs alone ranged fro 7 to 358; there were a total of 639 dierent stubs used. is the robability that the next coand is \dvis crossword.dvi", given that the revious coand was coand 7 = \latex crossword.tex", and this execution roduced an error. In general, of course Y j i = P ( Cd t+ = P j i j Cd t = coand i ; ErrorCode > 0 ); note that Y j j= 7 =. Siilarly, N j i = P ( Cd t+ = j i j Cd t = coand i ; ErrorCode = 0 ); A j i = P ( Cd t+ = j i j Cd t = coand i ; Tie = Afternoon ); W j i = P ( Cd t+ = j i j Cd t = coand i ; Day = Wednesday ); etc. In our exerients, we set n = 000 and = 0; this eans our encoding involved at ost n47 = ; 0; 000 values, but less for users who used under 000 coands. We show below that this is a robust rediction echanis that is (relatively) resource friendly. The rest of this section discusses how we estiated these values; the next section, how we used these quantities to redict the ost likely coand(s). Estiating the Probabilities: To roduce this \table", we need rst to dene which coands to include as \current coands" (i.e., the far left table in Figure 4), and as \redicted coands" (iddle colun), and second; to coute the actual conditional robabilities used to ll the values. As exlained above, as the distribution is not stationary, we should not sily use frequency estiates e.g., we should not estiate P j i as the nuber of ties we observed Cd t = c i followed by Cd t+ = c j (over all ts), divided by the total nuber of ties that Cd t = c i. We instead used the AUR, in several laces: First, as we can kee only 000 \current coands", we need to know which coands are still active. We therefore aintain robabilities for the various coands, and udate the using this technique. We then kee only the 000 coand lines with highest values. This eans we aintain coands used recently, and let the coands that are unused for an extended tie, fall away. It also eans new coands have a chance of being included, which would not haen if we, instead set the robability to the eirical frequency. For exale, iagine that the rst tie latex was used was in the 000 th coand. In the \robability eirical frequency" aroach, the robability of latex would be only =000. As this is robably the sallest value, it is likely the one that would be ushed! Note that this is robleatic, as this eans that the syste will never consider any new coands even if the next 00 coands are all latex! We also use this AUR to set and udate the conditional robabilitiy values P j i. For each current coand coand i, we then kee only the 0 redictions with the highest P j i scores. We also used this AUR to

6 Accuracy Cd? 0 Cd? 00 Stub? 0 Stub? 00 revious line 7.5% ****** 8.6% ****** revious 5 lines 38.8% ****** 60.7% ****** ost freq 5 lines 34.% 33.9% 6.% 6.3% fro coand 46.9% 47.4% 7.7% 73.3% fro coand + arsing 43.8% 44.0% ****** ****** fro coand + error 46.9% 46.9% 7.6% 7.6% fro coand + day 46.7% 46.7% 7.4% 7.5% fro coand + tie 46.6% 46.6% 7.4% 7.4% fro coand + day + tie 46.6% 46.6% 7.4% 7.4% using last coands 36.9% 36.9% 59.% 59.5% Legend: Cd Line \Colete Coand Line" Stub \Just coand itself (no args, otions)" Accuracy \to ve includes next coand line/stub" Table : Suary of results set the values for the Y j i and other values. (Note, however, that the decision of which redicted coands j j i to kee deends only on the Pi values, and NOT the Y j i quantities, nor does it deend on W j i, etc.) We need an value for each of these AURs; in our exerients, we set = 0.90 to decide which current coands to kee P = 0.95 to udate the Pi j 's Error = 0.9 to udate ErrorCode robabilities DoW = 0.90 to udate DayOfWeek robabilities T od = 0.90 to udate TieOfDay robabilities which we erically found gave the best results. 4 Exerients and Results This section resents our eirical results, based on the data fro the Greenberg (Gre88) dataset. Ex# rst attets to dulicate the Davison and Hirsh results, dealing only with stubs, albeit on our datasets. The rest of this section considers several extensions: Ex# argues for redicting colete coand lines, rather than just the stubs, and resents our results here. There are obvious ways to arse coand lines, and then re-use that inforation e.g., after \latex foo.tex", we ay exect the next coand to be \dvis foo.dvi -o"; note the foo is reeated. Ex#3 states this notion ore recisely, and then resents our results. So far, everything deals only with coand lines or stubs. We would naturally exect that our syste could ake better redictions if it were given ore inforation about the context of the current/revious coand. We therefore exlored using \error codes" (Ex#4) and \current day and/or current tie (Ex#5). Another source of inforation is the coand that was tyed before the current coand; Ex#6 discusses our attets to estiate and use P ( t+ j t ; t? ). Finally, Ex#7 and Ex#8 consider redicting the next coand stub, but here using (resectively) the revious coand line, and then the coand line lus the error code. The nal Ex#9 discusses the redictability of dierent classes of users; it also rovides additional inforation about the Greenberg dataset we are using. We suarize our ain results in Table. Before exlaining the details of this table (see below), note iediately the winner, across the board, is the silest aroach, of sily using the current cond in redicting the next coand i.e., argax P ( t+ j t ), using the AUR to aintain the distribution. In articular, we did not irove (and often, did not even atch) this accuracy when we also used error code, day, tie, or earlier coands. In ore detail: The \Cd?" coluns in Table deal with redicting the colete coand line, and \Stub?"'s, with redicting only the stubs. The = 0 coluns test the redictive accuracy over the user's EN- TIRE sequence of data; the = 00 coluns test the redictive accuracy after ignoring the rst 00 exeriences of each user. (Recall this is considered the algorith's learning hase, to hel it adjust to the erson.) 7 To get soe baseline values, we also ileented a nave rediction ethod that sily redicts the last 5 coands. This syste obtains 60.7% accuracy redicting coand stubs and 38.8% accuracy redicting coand lines. We get even worse nubers, of course, if we consider just the revious SINGLE coand. We also considered a syste that sily redicts the 5 ost 7 We chose 00 after nding it worked better than other training-set sizes, in the context of \learning full coand lines fro full coand lines" (i.e., the \winner"): Nuber ignored: Predictive accuracy: 47.3% 47.4% 46.7% 43.3%

7 frequent revious coands; these values were coarable. (They too aear in Table.) Clearly any rediction syste that erfors worse than these should sily be considered a failure. The following subsections focus on the redicting the coand line fro various bits of evidence, in the \raw accuracy" odel. The table resents the other data, dealing with the \accuracy?00" odel, and with the stub-rediction task. Ex#. Predicting Coand Stubs fro Coand Stubs. First, we sily dulicated the Davison and Hirsh algorith, and obtained 7.7% accuracy (accuracy?00 score of 73.3%). This is reasonably close to their \near-75%" accuracy. (Note that we are using a dierent dataset.) Ex#. Predicting Coand Lines fro Coand Lines. As noted earlier, redicting entire coand lines would be treendously ore useful that just redicting stubs. When we use the entire coand lines to redict the next coand line, our accuracy is 46.9%. (47.4% when we ignore the rst 00 \trainingcoands"). It is not surrising that this is considerably saller than the accuracy for redicting stubs, as this is a uch ore dicult task. (Recall we are only satised with a erfect atch.) Of course, getting the stub right ight still be useful, even if the rest of the coand line is incorrect. For exale, we could then give the user the otion of oving the stub into the user's buer, to be augented, by hand, with the aroriate arguents and otions. We therefore considered a dierent scoring easure, where we awarded the syste oint if the correct line aeared in the list resented to the user; 0:5 oints if the correct stub aeared, and 0 otherwise. Here, the average score rose to 57.8% that is, 46.9% of the tie our list included the correct line, and an additional.8%, it included the correct stub. Ex#3. Predicting Coand Lines fro Parsed Coand Lines. Most Unix coands follow the attern: hstubi hswitchesi harguentsi. By arsing the coand lines into a sequence of tokens, it is ossible to identify coon atterns between a coand that occurs at tie t and a coand that occurs at tie t +. (Treating hstubi hswitchesi as a distinct coand is reasonable, since any users alias coon hstubi hswitchesi airs (for exale, aliasing \la" to \ls -a").) It then becoes straightforward to coare the arguents between two coands and identify equivalent atterns. This can be further extended by dividing each harguenti into ha-stubi and ha-exti e.g., \foo.tex" becoes \foo" and \tex"). For exale, a coand sequence could be arsed as follows: Original vi foo.tex latex foo.tex dvis foo.dvi gv foo.s Parsed vi foo.tex latex harg i dvis harg? STUB i.dvi gv harg? STUB i.s Parsing the coand lines identies generalities that ay or ay not be correct. 8 For instance, after the shell had been trained on the above exale, if a user then enters \vi /etc/hosts" the redicter ay then suggest \latex /etc/hosts", which is not very likely to be correct. Further suose that \vi /etc/hosts" had occurred, followed by ing otherhost. In the odel we ileent, the shell can identify relationshis such as this uch ore readily if it does not arse the coand lines. By arsing the coand lines, we end u with a lossy data coression. The average accuracy for arsed coand lines is 43.8%. (accuracy?00 score of 44.0%). Aarently, the benets of knowing what to do when seeing \vi bar.tex" are outweighed by the loss of individual atterns. Ex#4. Predicting Coand Lines fro Coand Lines and Error Codes. All Unix coands have a return code, tyically an error code fro within the rogra. Moreover, note that the next coand is often be deendent on whether or not the revious coand succeeded. For exale, if coilation is successful, users will tyically execute the new object code. But if coilation fails, we ight exect the user to either run a debugger, or edit the source code. We therefore decided to include this, as art of the criteria for deciding on the roer action. As the eaning of the error code deends on the coand itself, we quantizing it down to no error versus error, (read \0 versus non-0"). This also ket anagable the size of the data structure, while roviding us with a reasonable aount of inforation (5.% of the coands returned an error). However, using the error code rovides no iroveent over the basic odel roducing an accuracy of 46.9%. Why? First, any coands do not (usually) return an error. Second, the total nuber of coands likely to follow a given coand is relatively low. As we were redicting the to ve coands, we found that the coands with a atching error code are likely to be listed, anyway (by erhas earlier). Ex#5. Using Day of Week and Tie of Day. We had anticiated that knowing the day of the week and/or the tie of day (orning, afternoon, evening, night) would be helful, guring that eole work in dierent odes during the daytie versus nighttie; or between week-days and week-ends. We found, however, 8 Note that this arsing inforation could also be useful if we later decide to generate scrits, as it can hel distinguish the \variables" fro the \constants".

8 Grou nae Predictability # subject #Cds non-rograers ,608 novice rograers ,43 exerienced gs ,906 couter scientists ,69 Table : Greenberg's DataSet that it did not hel in general: as the table shows, this inforatoin caused the average accuracy to go down slightly by about 0.5%. This ay be because the granularity used is too coarse and so this coutation ends u dulicating P ( t+ j t ). Ex#6. Predicting Coand Lines fro Multile Coand Lines. Identifying a long-ter trend for redicting the next coand ight be ore eective. By using the last two coands, we get 36.9% accuracy for coand lines and 68.8% accuracy for coand stubs. (Here, we just ket the best n = 000 revious-coand-airs, and used the in the sae way we had used revious-single-coands.) Note that this coand line accuracy is worse than the nave aroach that sily redicts the revious ve coands! Ex#7. Predicting Coand Stubs fro Coand Lines. Above we sought ways to to irove on the rediction rate for coand lines. We ight be able to use soe of these ideas to irove the accuracy of redicting coand stubs. By using the additional inforation in a coand line, the list of ossible hcoand? stubsi ay be reduced (hoefully reducing the chance of error). Unfortunately, we then lose the generality for coands that have not occurred before, and the coand stub accuracy dros to 68.7%. Ex#8. Predicting Coand Stubs fro Coand Stubs and Error Codes. Although using the error code had relatively little iact on redicting coand lines fro coand lines, it causes coand stub rediction accuracy to dro slightly 7.6%. This suggests that the error code ight not contribute any ractical inforation with the current odel; it but ay still be useful in future work. Ex#9. Predictability of Dierent Classes of Users. As entioned earlier, the Greenberg data was originally used to learn to classify each couter user into one of the four categories shown in Table. We see clearly that novice rograers are by far the ost redictable, and couter scientists, the least. 9 A 9 It is interesting to seculate on why: novices robably know relatively few coands (which ilies a saller set of coands to redict fro); and couter scientists are always trying out new things and in articular, new coand and new sequences. later syste ay be able to exloit this inforation, in heling to set u a user role, or whatever. (The table's two coluns indicate the break-down of the data: how any users were of each categories, and how any coands, total, were fro each category.) 5 Contributions, and Future Work Our results, suarized in Table 4, reinforce the Davison/Hirsh clai that it is ossible to redict user actions, using a fairly sile and ecient algoriths. Moreover, the accuracy is high enough to lead to a ractical, usable syste. While the accuracy is not as high as we would like, it has roven surrisingly dicult to irove on the accuracy score. Perhas this shows that we have, in fact, achieved the inherent redictability of the data i.e., eole truly are rando (to this degree), or at least, we aear to be, given the available data. This last thee, in turn, suggests that we should consider other sources of inforation, to better cature the context. For exale, any users use ultile windows siultaneously (e.g., one for editing and one for coiling). Moreover, today's window anagers (e.g., fvw) allow users to execute alications without tying. Obtaining inforation about when these events occur is also iortant. Finally, external events have an iact on what a user will do. Identifying events such as eail arrival, or high load averages, should also be considered. Of course, our long-ter goals are not sily saving keystrokes for the sall oulation of eole that use Unix-style shells. Instead, our goal is to obtain a better understand of techniques for redicting future user interactions, in the hoe of using such technologies to irove other interfaces erhas for coon software roducts such as Word or PowerPoint. We susect that we will be able to use these, or related, techniques not just to redict the coands, but also to detect atterns of the user interactions, and use this to rovide a truly helful assistant uch in the line as the Microsoft \Oce Assistant", but one that is adative to the individual users. (E.g., that can deterine when the user doesn't want to be bothered, or that soe secic user tyically enjoys hearing new hints, or... ) Sile, yet owerful, interfaces are clearly iortant today, given the vast nuber of interactive alication rogras. As these alications scale u, eective interfaces ay becoe even ore essential. We anticiate that adative interfaces, caable of roducing interfaces that users will be willing to use, will be a ajor tool used in building eective interfaces. We hoe the results resented in this aer will hel future researchers better focus on the relevant asects of this task. Finally, any reader who wishes to be art of the subsequent studies (and then to one of

9 the rst to rea the eventual benets of this adative shell) should read the aterial in htt:// Acknowledgeents We would like to thank Saul Greenberg for his dataset, which recorded traces of 68 users; without that data, our ain study would not have been ossible. Portions of this aer were adated fro work done with Thoas Jacob. Greiner was suorted, in art, by NSERC. References [BD77] Peter J. Bickel and Kjell A. Doksu. Matheatical Statistics: Basic Ideas and Selected Toics. Holden-Day, Inc., Oakland, 977. [DH97] Brian D. Davison and Hay Hirsh. Toward an adative coand line interface. In Advances in Huan Factors/Ergonoics, ages 505{508. Elsevier, 997. [DH98] Brian D. Davison and Hay Hirsh. Predicting sequences of user actions. In Predicting the Future: AI Areaches to Tie-Series Analysis, ages 5{. AAAI Press, July 998. WS [Gre88] Saul Greenberg. Using unix: Collected traces of 68 users. Research Reort 88/333/45, Deartent of Couter Science, University of Calgary, Calgary, Alberta, 988. [HBH + 98] Eric Horvitz, Lack Breese, David Heckeran, David Hovel, and Koos Roelse. The luiere roject: Bayesian user odeling for inferring the goals and needs of software users. In UAI-98, ages 56{65, July 998. [Lan97] Pat Langley. Machine learning for adative user interfaces. In Geran Articial Intelligence, ages 53{6, 997. Gerany: Sringer. [SH93] J. C. Schlier and L. A. Herens. Software agents: Coleting atterns and constructing user interfaces. JAIR, :6{89, Noveber 993.