Identifying and Understanding Dates and Times in

Transcription

1 Identifying and Understanding Dates and Times in Mia K. Stern Collaborative User Experience Group IBM Research 1 Rogers Street Cambridge, MA mia_stern@us.ibm.com ABSTRACT is one of the killer applications of the internet, yet it is a mixed blessing for many users. While it is vital for communication, has become the repository for much of a user s important information, thus overloading with other responsibilities. One way in which users overload their is by using it as a calendar and a todo list. People keep in their reminders of meetings, events, and things to do by some deadline, all of which contain dates and times. Unfortunately, these s can get lost amongst all the other s. Because of the numerous ways dates and times can be expressed in written language, traditional searches are often not effective. In this paper, we discuss a technique to help users extract such calendar and deadline information from their s by identifying dates and times within an message. We believe identifying dates and times will help users organize their schedules better and find lost information more easily. In this paper, we discuss our approaches for this problem. We discuss syntactic methods used to find dates and times and semantic methods to understand them. We then present the results of two user studies conducted to determine the accuracy of our technique. Keywords Date extraction, Time extraction, , Evaluation INTRODUCTION Increasingly, people are using their inboxes as a way of organizing their lives (Duchenaut & Bellotti, 2001)(Whittaker & Sidner, 1996). Inboxes are no longer simply repositories of incoming mail; they are where the details of peoples' lives reside. People use their inboxes as calendars, to-do lists, and address books, among other things. People keep documents in their inboxes because those documents have pieces of information that they do not want to delete. They also save documents to keep the information readily accessible. Unfortunately, the more documents that accumulate in the inbox, the harder it is to manage the information that is contained there. While documents are semi-structured in that they contain well-defined fields, the bodies and the subjects are unstructured. These unstructured parts may contain some information that can help the user organize important information better, such as names of people and companies, URLs, phone numbers, and places where meetings take place. documents also contain dates and times that allow users to use their inboxes as calendars and to-do lists. In this paper, we focus on extracting this date and time information from documents to make some user's tasks easier. We have developed an add-on to (currently implemented in Lotus Notes) that can help users keep track of items in their inboxes by taking advantage of the semi-structured information provided by dates and times. This system identifies date and time phrases that appear in the bodies and subjects of messages, and interprets these phrases into a canonical calendar format. There are a number of potential applications for this technology. For example, the system can help the user more easily make calendar entries (such as appointments and meetings) and to-do items. When the user wants to make such an entry, she can choose from the dates and times found and the entry will be made for that selection. Nardi, et.al. (Nardi, et al., 1998) present a similar idea to this, but in their system, the user must first select the text which contains the date to be parsed. Our system also allows this functionality, but it can also find all dates and times in an message without requiring the intervention of the user. This technology can also be used for smart reminders, indicating messages with an approaching due date, or even messages that have expired. Furthermore, users can search through their for a date, regardless of its textual format. This paper focuses primarily on the accuracy of the underlying text extraction techniques that will support a range of such applications. The rest of the document is organized as follows. We begin by discussing our technique for locating dates and times within an message. We then describe the methods for semantically processing a date or time. The results of our user studies are then discussed. We conclude with some thoughts and future work.

2 FINDING DATES AND TIMES The goal of this project is to identify and understand dates and times that appear within messages. Although there has been previous work on identifying dates/times in standard corpora (Mani et al., 2001)(Grover et al., 2000)(Message understanding conference), in historical documents (Mckay, 2001), and in scheduling dialogs (Wiebe et al., 1998), there has not been a focus on the kinds of dates/times that appear in messages. The Selection Recognition Agent (Pandit and Kalbag, 1997), an application-independent feature recognizer, could be applied to , although that was not the main focus of that work. The purpose of this testing was to build a lightweight date/time extractor to start building applications specific to . The first step was constructing a grammar that would detect date and time phrases in messages. For this, we constructed regular expressions to find the dates and times, since regular expressions are a simple way to represent most of the date and time expressions we have discovered in messages. Some of the regular expressions we are using can be found in Figure 1. The seventh regular expression, MONTH_DAY_YEAR, can identify, for example, January 1st, The last expression, TIME_AMPM, can identify times such as 10:03 a.m. SHORTMONTH = (Jan Febr? Mar Apr May Jun Jul Aug Sept? Oct Nov Dec)\.? LONGMONTH = January February March April May June July August September October November De cember MONTH = SHORTMONTH LONGMONTH DAY = [0-2]?[0-9] 3[0-1] YEAR = \d{4} SUFFIXES = st rd th nd MONTHDAYYEAR = MONTH\s+DAY+SUFFIXES?,?\s+YEAR HOUR_12_RE = 0?[1-9] 1[0-2] HOUR_24_RE = [0-1]?[0-9] 2[0-3] MINUTE_RE = [0-5][0-9] AM_RE = (a (\s*am) (\s*a\.m\.)) PM_RE = (p (\s*pm) (\s*p\.m\.)) AM_OR_PM_RE = AM_RE PM_RE TIME_AMPM = HOUR_12_RE(\s*:\s*MINUTE_RE)?\s*:\s* MINUTE_RE AM_OR_PM_RE Figure 1: Example regular expressions One of the most challenging parts of this project was defining qualities of dates and times to test for that would anticipate the needs of a set of meaningful and calendar applications. These included figuring out when to present a date to the user as a single date and when as a range of dates, and how to define certain ranges. Single dates, occur on only one day whereas date ranges cover multiple days. We have determined a similar classification for times, where single times refer to a moment on a calendar and time ranges have distinct start and end times. Furthermore, both dates and times can be classified as explicit or inexact. Exact dates and times are those that are explicitly spelled out, while inexact dates and times are more vague in their references to a calendar. Inexact dates are typically in reference to a known date, and inexact times have a more fuzzy start and end than do explicit times. Table 1 shows some examples of dates and times that fall under these categories. Our use of regular expressions has some limitations. While regular expressions can be very fast and reasonably accurate for date and time detection, they are not sufficient for finding all dates and times, as there are a large variety of formats for those expressions. Similarly, they are not sufficient for linking a related date and time that are split by extraneous text. For example, in the phrase January 24, 2002, 1 Rogers Street Room 5003, 12:00-1:30pm, it is very hard to link January 24, 2002 with 12:00-1:30pm. Our goal, however, is not perfection with regards to feature detection, since heuristics are by their nature imperfect. Rather, we are hoping to be close enough so that users will find this technology useful and usable. Dates Explicit Inexact Single 8/16/ August 2002 Tomorrow Next Thursday Range August 8-12, 2002 June 12 July 5 Next week August 10am noon Explicit 10am from 3 to 4pm Times Morning Inexact At 4 lunchtime Table 1: Examples of different kinds of dates and times Understanding dates and times Once a date/time has been located, it must be semantically parsed so that it can be used by the applications we have mentioned. To do this, the system must convert each date/time found into a canonical format. In this section, we discuss the heuristics we use for this semantic analysis.

3 Kind of expression No year given This <day> Next <day> Last <day> No a.m. or p.m. Heuristic assumption Check verb tense, if future assume within next 12 months Before the end of this week During the week that starts on the upcoming Sunday During the week that ended on the previous Saturday Assume during normal business hours Example We will meet on February 4 Date/ time interpretation for message received on January 28, 2003 February 4, 2003 This Thursday January 30, 2003 Next Thursday February 6, 2003 Last Thursday January 23, 2003 Let s meet at 1 Table 2: Some heuristics used for filling in missing fields January 28, 2003 at 1pm Missing fields Dates and times that are fully specified are easy to convert into this canonical format. A fully specified date is one that has its calendar date(s) and time(s) explicitly stated. An example of a fully specified date/time is Thursday, March 28, 2002 from 1:00pm to 2:00pm. These kinds of dates tend to appear in formal meeting announcements or talk announcements. However, many s, especially those written using a more conversational tone, do not contain such formal dates and times. Dates and times occurring in messages are more informal, often assuming human readers can fill in the unspecified portions. For example, people can easily process and understand Let's meet at 4 on Thursday by using their background common sense knowledge and experience. In our system, heuristics are needed to fill in the missing fields of both dates and times. For example, when writing times, people will frequently omit a time of day indication (a.m. or p.m.). If this omission occurs within , we assume the hour referred to occurs during the regular business day, i.e. 7am through 6pm. People will also write times without associated dates, such as Let's meet at 10am. In this case, we assume the date to be the reference point date (described in the next section). With dates, different details can be left unspecified. A very commonly unspecified detail for a date is the year, e.g. March 29. In this case, we assume the date to be within the next twelve months of the reference point date (March 29 would be as March 29, 2003). Inexact dates frequently need many of the date/time fields filled in. These kinds of dates can be confusing to interpret, for human readers as well as computers. For example, if today is Monday, and the document mentions next Thursday, is the Thursday in question the day 3 days from the reference date or 10 days from that date? We are making the assumption that next anything is within the week that starts from the upcoming Sunday and ends the following Saturday. A sampling of some of the heuristics used can be found in Table 2. Setting reference point dates for inexact dates Inexact date phrases and time-only phrases need a reference point from which the date for the phrase can be calculated. Some systems, such as LookOut (Horvitz, 1999), use the date the message was sent as this reference point. However, this is not always correct. For example, in Figure 2, there are two instances of the word tomorrow. If we were to use the date the was sent as the reference point, the second tomorrow would be incorrectly. messages, however, provide additional clues as to what these reference points should be, namely those dates which can be found in headers within the body of an message. Messages which are replies or forwards frequently contain these kinds of textual headers. Reference points are determined by treating the as a series of headers and bodies, with each header and body making a header block. The header of the message starts the first header block, and subsequent headers found within the body start their own header block. Each block is terminated by the subsequent header. Each header contains the date indicating when that part of the document was sent. The system uses this date as the reference point for any dates or times found within the header block.

4 Using this new heuristic, both of the instances of tomorrow in Figure 2 would be correctly. The first is as May 18, 2002, while the second is as May 15, From: Mia Stern 05/17/ :12 PM To: Derek Lam cc: Subject: Re: Meeting Sorry I couldn't make it then. How about tomorrow instead? - Mia From: Derek Lam 05/14/ :38 PM To: Mia Stern cc: Subject: Meeting Figure 2: Why headers are used for reference point dates, rather than using the sent date of the message Part of speech tagging Can you make a meeting tomorrow? Derek We use another heuristic technique for determining the correct date being referred to in a document. We cannot always assume that dates are in the future. If we make such an assumption, we cannot process phrases like As we said on Thursday and accurately determine the correct date. Therefore, we use part of speech tagging (Brill, 1992) to determine if the date is in the past or the future. We look at the verb closest to the match, and if it is in the past tense, we assume the date is also in the past. A similar approach to using parts of speech is presented in (Mani & Wilson, 2001). From: Mia Stern 01/15/ :43PM To: Daniel Gruen cc: Subject: web site Hi Dan- In the meeting we had on Monday, we talked about setting up a new web site. Can we meet on Friday to talk about this more? Thanks, - Mia Figure 3: Using part of speech tagging for disambiguating the meaning of a date phrase. For example, consider the message given in Figure 3. Without part of speech tagging, Monday would have been as Monday, January 20, However, since the closest verb to that phrase is in the past tense (in this case had ), the phrase is correctly as Monday, January 13, Similarly, since the verb closes to Friday, meet, is not in the past tense, that phrase is to mean Friday, January 17, Ambiguities There is another difficulty with the semantic processing of dates and times. There are ambiguities that arise with dates and times that can be difficult to resolve without context. For example, is Thursday, 7-10 a date, as in July 10th, or is it a time, from 7pm to 10pm (or 7am 10am)? Our current method is to assume in this case a time, rather than a date. In the ideal case, both options and context surrounding the match would be presented to the user so she could disambiguate more easily. EVALUATION Before exploring whether any applications using this date/time understanding technology would be viable, we wanted to investigate whether the technology could accurately identify and interpret the specific dates and times that appear in messages. We also wanted to discover if there were any differences in how users rate the dates and times that were found and how they were. Methodology We conducted a user study with 9 participants in which each user processed about 20 of her own s. Each user was presented with her one message at a time, and for each message, she is asked about the dates and times that were found. Each date/time found is presented one at a time and a series of questions appropriate to the kind of date/time is asked. The first question asked is obviously, Is this phrase a date/time related phrase? If the user answers yes, she is then asked if the date is classified correctly as a single date or a date range, if that classification is correct, and if the date(s) is correct? The user is then asked about the time classification and the time interpretation. We present our results in terms of precision and recall. Precision is the number of date/time phrases correctly processed in category x divided by the total number processed in category x (these can also be called false positives). Recall is the number of date/time phrases correctly processed in category x divided by the number of things that really should have been processed in category x. See Table 3 for an illustration of how precision and recall are calculated.

5 Is this a date/time related phrase User Identifies as Date/Time User Rejects as Data/Time System Proposes as Date/Time A (correct) C (false positives) Precision = A/(A+C) System Does not Propose B (misses) Recall = A/(A+B) Table 3: How precision and recall are calculated. The grayed out box cannot be calculated. Results We collected 150 messages from the 9 users in our study. Our user population consisted of interns, developers, researchers, and one executive. Of the 150 collected messages, 78% had dates and/or times. Eleven of the 117 messages contained only machine generated dates (e.g. dates that appear within header blocks). Within these 117 messages, our system proposed 593 date/time phrases, of which 546 were actually date/time phrases. Thus our precision is 92.07%. Our system missed an additional 39 dates/time phases that users identified, giving us a recall value of 93.33%. 1 These are just broad claims, however, indicating whether the system is focusing on the right types of phrases. To be effective, the system must also correctly classify the type of date/time phrase and accurately interpret the date or time found. In the sections that follow, we delve into more detail on how well the system classified and single dates and date ranges and then how well it did on both single times and time ranges. Dates Once a phrase has been identified as a date/time phrase, it must be classified as either a single date or a date range. On single date identification, the system achieved a precision of 89.65% and a recall of 92.92%. With date range identification, on the other hand, the precision is only 74.82% while the recall is only 83.2%. Clearly the system does not perform as well on identifying date ranges. However, there are almost 4 times as many single 1 It is possible that the documents we collected are skewed in the number dates and times, since users analyzed only those documents they wished to share; we do not know the proportion of date and time phrases in documents users did not allow us to see. dates as date ranges, indicating that perhaps single dates are a more important focus for this kind of work. Table 4 illustrates how well our system performs on both classifying and interpreting single dates. We have broken down the kinds of date phrases into whether they were an explicit date, an inexact date (e.g. tomorrow, next Tuesday, or this morning), a time without a date (e.g. 1pm or from 1-2pm), and all others. The rows in the table indicate the user responses, including whether the date was classified and correctly, whether it was either misclassified (the system claimed the phrase was a single date when it was not) or incorrectly (the system knew it was a single date but it got the date wrong), or whether it was not really a date at all. Overall, the system achieved 81.06% precision on interpreting single dates and 84.02% recall (the system missed 28 single dates and claimed that 3 actual single dates were date ranges) on locating and interpreting single dates. We expected the system to be able to interpret explicit dates relatively well but to have difficulties with other kinds of dates. As anticipated, the system had more difficulty classifying and interpreting inexact dates than explicit dates, since it needed to infer the date from context rather than just parse the phrase. It also had some difficulty associating the correct date with phrases which contained only a time. One possible reason for this difficulty is due to the heuristic we are using for filling in the missing date in those cases. We are currently using the reference point date of the header block, which is frequently not the correct date. However, if we change our heuristic to use the closest date to the time phrase in the sentence, we feel we can improve the accuracy on those phrases. In Table 5, we see how well our system does on identifying and understanding date ranges. Similar to our analysis of single dates, we have broken down date ranges into various kinds of ranges, including explicit ranges (e.g. November 6-8), months with years (e.g. June 2003), months without years (e.g. June), years or year ranges (e.g or ), and inexact date ranges (e.g. this week or next week). The rows are the same as in Table 4. Overall the precision for interpreting date ranges is 69.06% and the recall is 82.05% (the system missed 10 ranges and claimed 11 actual ranges were single dates). It is interesting to note how many kinds of date ranges there were, with most of them being somewhat ill defined, and only 5 explicit date ranges. There were a large number of inexact ranges, and the system did relatively well on those kinds of dates. The largest number of date ranges were years or year ranges. However, one reason the system performed poorly in general on date ranges is the number of phrases the system classified as years that users did not consider date ranges at all. One reason for this is our regular expression for detecting years is overly general; it detects every four digit number. By limiting the range for years to between 1900 and 2099, we should

6 Correctly classified and Incorrectly classified or Not a date phrase Specific date 196 (95.6%) Extraction process: Single dates Inexact date 127 (74.27%) Time only Other total 44 (64.71%) 1 (10%) 368 (81.06%) total Table 4: System's performance on identifying and interpreting single dates. The percentages given are precision values, calculated by dividing the value in the cell by the total for that column. Correctly classified and Incorrectly classified or Not a date phrase Explicit date range 5 (100%) Month w/ year 11 (91.67%) Extraction process: Date ranges Month w/o year 9 (81.82%) Year / year ranges 44 (57.14%) Inexact date range 26 (78.79%) Other 1 (100%) Total 96 (69.06%) Total Table 5: System's performance on identifying and interpreting date ranges. The percentages given are precision values, calculated by dividing the value in the cell by the total for that column. be able to reduce the number of false positive year detections. By making this change, we would have avoided 21 false positive instances, increasing precision on date ranges to 81.36%, up from 69.06%. Overall, on dates, the system achieved a precision value of 78.25% and a recall result of 79.32%. We believe that with some simple changes to some of our heuristics, we can potentially improve these results. Times In addition to dates, we also investigated how well the system could find and interpret times. Table 6 shows the system s results on finding times associated with dates. An interesting thing to note from this table is the frequency with which the system missed times. Many of these missed times occur when a date and time appear within the same sentence but are located far apart, similar to the problem we discussed with incorrect dates being associated with time only phrases. For example, let s meet tomorrow, say around 1pm contains two date/time phrases, tomorrow and 1pm. Our system misses the time for tomorrow and assigns the incorrect date to 1pm. By using the heuristic discussed in the last section, we hope to alleviate both problems. For single times, the system was very accurate (precision of 91.5% and recall of 80.63%). For these times, there were 112 instances when the time was given with either a.m. or p.m. The system correctly all 112. For the 17 instances in which no a.m. or p.m. was given, the system all but 2 correctly. In both cases, the system mistook a day for a time (e.g. in 01 JUL 02, it the time as 1pm). With time ranges, the system achieved a precision of 75.36% but a recall of only 71.25%. However, of the time ranges the system found, it correctly all 40 explicit ranges. For inexact ranges (such as morning or afternoon), the system correctly only 12 out of 26 cases. We believe this is because our definitions of

7 when these inexact ranges begin and end do not agree with users opinions on these ranges. Correctly classified and Incorrectly classified and/or Not a time phrase No time 171 (83%) Extraction process: time Single time 129 (91.5%) Time range 52 (75.4%) Total 352 (84.6%) Total Individual differences Table 6: Results for finding times. The results reported thus far are from pooling the data from all 9 participants. However, this kind of analysis does not help us determine if there are any individual differences in how users interpret dates and times. To determine if there are such differences, we performed an analysis on the accuracy of the date detection and understanding. We calculated the log odds 2 of the precision for each document a user evaluated, and then calculated the mean of these values and the standard error of the mean for each person and their documents (see Figure 4). We can see from this analysis that there no detectable difference between the means for the individuals and the mean for the population. However, based on discussions with users, we are inclined to believe there are some individual differences that cannot be detected statistically. For example, some participants did not agree with our specifications for the beginning and end of a week (in phrases such as this week or next week). Three participants consistently challenged our definition of when a week starts and ends, while the other six agreed with our specifications. With respect to time, the only occasions on which there were some disagreements were in how times of day should be, such as morning, afternoon, and 2 Log odds is calculated using the following formula: numcorrect ln( ). The 0.5 correction is a numfalsepositive Bayesian technique that may be used when the actual number of observations is small, e.g., one of the numbers might be zero. evening or night. Two participants disagreed about the start and end time for afternoon while one participant disagreed about morning. One possible approach to this is to model individual user preferences on these time ranges. Mean +- 1 SE USER1 USER2 USER3 USER4 USER5 USER6 USER7 USER8 USER9 Figure 4: Comparing means of log odds between users. The dotted line shows the mean for the whole population and the surrounding box is one standard error of the mean. Second user study We took the results from the first user study and analyzed where we could make improvements in our algorithm. There appeared to be some clear areas where we could make improvements without significantly changing our strategy. Some of the improvements were designed to increase precision while others should help improve recall. We made these changes to the algorithm and ran a second user study to determine if the results in fact improved. Rather than running our algorithm over the data we collected in the first user study, we decided to run a second study to determine if our results would generalize over a different data set. Improvements One improvement to increase precision is to restrict our definition of a year. In the first user study, we were detecting all 4 digit numbers as years. This led to a large number of false positive finds. To reduce the number of false positives, we are now only recognizing years between 1900 and A second improvement to increase precision is a heuristic to link dates and times together. During the first user study, the system frequently did not find times associated with some dates when the participant indicated that there was in fact a time. Similarly, on many phrases that consisted of only a time or a time range, the system assumed an inappropriate date for that phrase. To fix this problem, the system now looks within a sentence to see if there is a time to associate with a date or a date to associate with a time. However, if there is more than one

8 Correctly classified and Incorrectly classified or Extraction process: single dates (second study) Specific date Inexact date Time only Total 193 (88.53%) 128 (89.51%) 26 (26.53%) 347 (75.60%) Not a date phrase Total Table 7: Single date results from the second user study. Correctly classified and Incorrectly classified or Explicit date range 3 (100%) Extraction process: date ranges (second study) Month w/o year 9 (69.23%) Year or year range 24 (40%) Inexact date range 22 (91.67%) Deadline date 42 (85.71%) Total 100 (67.11%) Not a date phrase Total Table 8: Date range results from second user study. Correctly classified and Incorrectly classified or Extraction process: time (second study) No time Single time Time range Deadline times Total 185 (93.91%) 129 (96.27%) 60 (57.14%) 46 (92%) 420 (86.42%) Not a time phrase Total Table 9: Time results from the second user study. time in a sentence, the system will not choose one to match with a date, and vice versa. To improve recall, we are now also detecting date and time ranges we are calling deadline dates and times. These phrases are distinguished by including keywords such as by, until, or even through. Previously, we were finding the dates and times in these phrases, but we were not recognizing that those dates and times indicated a range, rather than a single date or time. For example, in the phrase The paper is due by October 14, we would find October 14, but we would not recognize that this date represented a deadline. Users indicated that the whole phrase should be identified as a date range, starting when the was received and ending at the date in the phrase. The last improvement we made, also to improve recall, is to expand our notion of time ranges by detecting mealtimes, such as lunch(time) or dinner. We are also now detecting the phrases AM and PM as time ranges. A number of users indicated during the first user study that these were phrases that represented times.

9 Results For the second user study, we had 7 users evaluate a total of 162 documents, 115 of which had dates. Our system detected 608 date/time phrases, 519 of which were actually date/times (precision = 85.36%). The system missed only 19 date/time phrases (recall = 96.47%). Our overall results for the second user study are fairly similar to the first user study (see Tables 8-10), with the only significant difference on precision being a lower precision in the second user study on single dates (t=2 on an independent samples t-test, p < 0.05). For recall, we increased our results significantly for single dates (t=3.874, p < 0.005) and for times (t=3.566, p < 0.005). What we are really interested in discovering, however, is the effects our specific changes made in these results. The first improvement we made, reducing the range of years detected as dates, lowered the number of false positives. In the first user study, 59.6% of false positive date detections were years, while in the second user study, only 35.2% of false positives were years. The second improvement we tried is to link dates and times that appear in the same sentence. Unfortunately, our heuristic did not apply in any instances in this dataset. However, there were still 23 cases in which a phrase contained only a time and the system could not correctly calculate the date. Similarly, there were 12 date phrases for which the system found no time but users indicated there was an associated time. In some of these cases, the dates and times were in adjacent sentences; in others, there were multiple times and/or multiple dates, which we specifically chose not to match up. Clearly we need another heuristic to help in these cases. Our third heuristic improvement was to look for deadline dates and times. These are phrases that start with through, until, or by. Our system found 50 deadline dates and times, only one of which was considered not a date phrase. The system achieved a precision of 84% identifying and interpreting these phrases. Thus it appears that adding deadline dates and times provides a good improvement to our algorithm. Finally, our last improvement, expanding our definition of time ranges, appeared to have a large negative impact on our results. Only 2 out of 27 instances (0.07%) where AM and PM were detected were considered to be legitimate time phrases, whereas 28.4% (25 out of 88) of our false positives involved these phrases. With respect to meal times, 9 instances were considered not to be date phrases and 9 instances were considered to be date phrases. Therefore, it seems in these cases, the false positive rate is too high to provide significant benefit from identifying these time ranges CONCLUSIONS In this paper, we have presented a technique to detect and interpret dates and times within . We have attempted to determine the accuracy of our technique through two user studies. We have demonstrated that we can achieve a fairly reasonable accuracy (about 80%) on finding and interpreting dates and times that appear in messages. We can detect and understand not only standard dates, such as July 1, 2003, but also inexact dates and times, such as next Tuesday morning. We are beginning to explore the use of this technology in some applications, including smart calendar entries and smart reminders. Additionally, we have built an application that lets users search for dates, regardless of the format of the search query or of the dates that are contained in the documents. However, we must determine if the accuracy we have achieved is good enough for these applications. There are certain classes of dates and times that we are currently unable to detect and understand: repeated dates. For example, Let s meet every Thursday at 10 indicates that a meeting will occur every Thursday at 10am. Currently, however, we do not recognize that the word every indicates a repeated event, not a one time occurrence. We plan to include these kinds of dates and times in our next version. ACKNOWLEDGEMENTS I would like to thank John Patterson and Daniel Gruen for their help in designing the study and in analyzing the data. I would also like to thank all the study participants for their time and patience. REFERENCES Brill, E., A simple rule-based part of speech tagger. In Proceedings of Third Conference on Applied Natural Language Processing, Trento, Italy, Duchenaut, N. and Bellotti, V., as habitat: An exploration of embedded personal information management. ACM Interactions, 8(1):30-38, September- October Grover, C., Matheson, C., Mikheev, A., and Moens, M., LT TTT A Flexible Tokenisation Tool. In Proceedings of Second International Conference on Language Resources and Evaluation, Horvitz, E., Principles of Mixed-Initiative User Interfaces. In Proceedings of CHI '99, ACM SIGCHI Conference on Human Factors in Computing Systems, , Mani, I., Ferro, L., Sundheim, B., and Wilson, G., Guidelines for Annotating Temporal Information. In Proceedings of the Human Language Technology Conference, 2001.

10 Mani, I. and Wilson, G., Robust temporal processing of news, in Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, Mckay. D., Mining dates in historical documents. In Fourth New Zealand Computer Science Research Students Conference, Message understanding conference. Nardi, B., Miller, J., and Wright, D., Collaborative programmable intelligent agents. Communications of the ACM, 41(3):96-104, March Pandit, M. and Kalbag, S., The Selection Recognition Agent: Instant access to relevant information and operations. In Proceedings of Intelligent User Interfaces, pages 47-52, ACM, Whittaker, S. and Sidner, C., overload: Exploring personal information management of . In Conference Proceedings on Human Factors in Computing Systems, , Wiebe, J., O'Hara, T., Ohrstrom-Sandgren, T. and McKeever, K. An Empirical Approach to Temporal Reference Resolution. Journal of Artificial Intelligence Research, 9, , 1998.