stanford hci group / cs376 Capture & Access Scott Klemmer 14 November 2006

Size: px

Start display at page:

Download "stanford hci group / cs376 Capture & Access Scott Klemmer 14 November 2006"

Shanon Park
6 years ago
Views:

1 stanford hci group / cs376 Capture & Access Scott Klemmer 14 November

2 M2 Exercise: Study Design 2

3 3

4 Tools and techniques to help field biologists collect, organize, and share their research content. 4 ButterflyNet IS a suite of tools and techniques to help field biologists collect, organize, and share their research more specifically, ButterflyNet contributes techniques to. 4

5 ButterflyNet System Capture Structure Access Transform Share 5 Capture and Structure content in the field and then access, transform, and then share their content when they are back at the field station. 5

6 Capture Structure Access Transform Share 6 It enables biologists to capture paper notes, photographs, and physical specimens 6

7 Capture Structure Access Transform Share 7 It provides three field techniques to structure content so that it can be 7

8 Capture Structure Access Transform Share 8 More efficiently accessed later using ButterflyNet s browser. Content from the browser can be exported to applications such as the 8

9 Capture Structure Access Transform Share 9 Multimedia Spreadsheet, which helps users transform data for analysis. The content can also be To Spreadsheets or Word Processors 9

10 Capture Structure Access Transform Share 10 Shared so that others can access your research from their browsers 10

11 Fieldwork studying field biologists Interaction Techniques mobile capture and access Evaluation first-use and longitudinal studies Current and Future Work integrated paper/digital interactions 11 The rest of this talk will be divided into four sections First, I will describe our fieldwork, and how our findings informed our designs. Second, I will describe the specific interaction techniques in ButterflyNet. Third, I will explain our first-use study of ButterflyNet, and touch upon current long-term studies Finally, I will briefly describe our current directions for further integrating current paper-based practices with computer support. 11

12 Interviews and Observations Qa 12 We began this work by interviewing 23 field biologists at Stanford University, the Jasper Ridge Biological Preserve, and the California Academy of Sciences visiting and observing people at their labs, {next} and in the field, and examining 13 of their paper notebooks (which we found was a central tool in current practices). 12

13 California Academy of Sciences 13 In visiting the California Academy of Sciences, we observed how some field research was archived, such as this notebook from Tracey Storer s field research in

14 Los Tuxtlas Tropical Rainforest 14 We visited tropical rainforest, following a field research class comprising 9 students, two TAs, and a professor. I observed how field experiments were run {next}, and how physical specimens were collected. 14

15 White Mountain Research Station 15 I ve also seen multiple research stations, such as the one in White mountain, CA to observe how field stations supported research, both with paper-based visualizations and with {next} remote monitoring 15

16 Jasper Ridge Biological Preserve 16 Finally, instead of just watching biologists I joined the Jasper Ridge Biological Preserve to become trained as one. I m currently involved with a camera trapping project there. 16

We observed that Transformation Capture is

17 Over the 350 hours of fieldwork we learned

17 We observed that Transformation Capture is (relatively) easy, but Access Sharing are not. 17 Over the 350 hours of fieldwork we learned many things but the most important thing that we observed, was that currently. Capturing content in the field using current tools is (relatively) easy. However, accessing, transforming, and sharing this content is not a simple process. 17

18 Current tools Pens & Paper Notebooks Cameras Spreadsheets Specimen Envelopes / Bags Laptop & Desktop Computers Sensors GPS Devices GIS Software Printers 18 Now if you look at the current toolset, you see pens, notebooks, cameras, spreadsheet software, envelopes for collecting specimens, printers, etc 18

19 Physical/Tangible Digital Pens & Paper Notebooks Cameras Spreadsheets Specimen Envelopes / Bags Laptop & Desktop Computers Sensors GPS Devices GIS Software Printers 19 And if you tease these tools apart, you see that some of these are primarily tangible, physical tools, while the others are digital, computer-enabled tools. The physical tools, such as inking pens, paper notebooks, and envelopes and bags, may seem to be relics of a past generation, but they are important because they support 19

20 Field tools are Physical Mobile Flexible 20 The field biologists current field practices. They are mobile, enabling biologists to hike long distances. They are physical and robust, and they support flexible input. 20

21 Different Advantages Paper Notebooks [Robust, Battery, ] Computers [Search, Storage, Sharing ] 21 Of course, there are many obvious advantages to paper notebooks. As we have noted from the two previous talks, paper is robust, it has infinite Battery Life. If you drop it it doesn t break. If you tear it in two, you get two paper notebooks. That is not the case with a tablet pc. However, as we are all technologists, we know that Computers offers fast search, vast storage, and the great capability to share with remote colleagues. 21

22 Combine Them! Paper Notebooks [Robust, Battery, ] Computers [Search, Storage, Sharing ] 22 So in this work, we sought to combine the two technologies. To bring the humble biologists tools into the digital age. 22

Digitizing Pen Technology 23 The way we combine the

by using Digitizing Pen Technology from Anoto.

with a camera that tracks a faint dot pattern that is

The advantage of this technology is that as you write

handwriting, as it knows on which page you are

importantly, the exact time that you are writing these

23 Digitizing Pen Technology 23 The way we combine the benefits of computing with the benefits of paper, is by using Digitizing Pen Technology from Anoto. This commercial technology comprises an inking pen, with a camera that tracks a faint dot pattern that is printed on a paper page. The advantage of this technology is that as you write inked notes on physical paper, the pen saves your handwriting, as it knows on which page you are working, where on each page you are writing, and most importantly, the exact time that you are writing these notes. This vector information is something that digital imaging (such as scanners or cameras) do not provide. 23

24 Fieldwork studying field biologists Interaction Techniques mobile capture and access Evaluation first-use and longitudinal studies Current and Future Work integrated paper/digital interactions 24 24

25 ButterflyNet System Capture Structure Access Transform Share 25 25

26 Capture 26 26

27 Three Techniques for Structuring Time-Based Assocation (Automatic) Hotspot-Association (Manual) Visual Specimen Tagging (Manual) 27 27

28 1 Automatic Association Notes + Photos associated by Time 4:43pm 4:44pm 28 28

29 2 Manual Hotspot Association Notes + Photos associated by Inked Gesture 29 BENEFIT: Explicit Linking 29

30 Gesture Recognition 30 30

31 3 Manual Visual Specimen Tag Association Notes + Photos + Specimens associated by 2dbarcode 31 31

32 Using Visual Specimen Tags photograph annotate collect annotation photos specimen 32 Used Gesture Recognition [Papier Craft] Now trying a more explicit interaction 32

33 ButterflyNet System Capture Structure Access Transform Share 33 [TODO: Screenshots of BNet around here ] 33

34 Access, Transform, and Share Access (Browser) Transform (Multimedia Spreadsheet) Share (Browser, USB, ) 34 34

35 ButterflyNet Browser 35

36 1 Automatic Association Notes + Photos associated by Time 36 36

37 37

38 38

39 39

40 2 Hotspot Association 40 Used Gesture Recognition [Papier Craft] Now trying a more explicit interaction 40

Upon viewing a photograph that happens to contain a barcode, the context panel will display any

41 3 Visual Specimen Tags 41 The results of the Visual Specimen Tagging looks like this in the ButterflyNet Browser. Upon viewing a photograph that happens to contain a barcode, the context panel will display any annotations that have been written on the envelope. We are currently working on ways to retrieve photos by presenting the barcode to a web camera as a querying technique. 41

The user invokes the query by tapping the pen to the physical page. Navigate by Pen 42 And finally, the ButterflyNet Browser provides a way for a user to navigate her digital notes using her pen.

42 The user invokes the query by tapping the pen to the physical page. Navigate by Pen 42 And finally, the ButterflyNet Browser provides a way for a user to navigate her digital notes using her pen. She can flip to the right page in her physical notebook, and then tap on the page with her pen. In real time, the notes browser will flip to the correct page, and show the new set of related photos. {next} Thus, you can use the physical notebook as a query mechanism to find related photos. 42

43 Multimedia Spreadsheet 43

44 44

45 45

46 46

47 47 47

48 48 48

49 49 49

50 50 50

51 51 51

52 52 52

53 Currently, in this system, we do not leverage handwriting recognition, as this method is flexible and supports a biologist s personal abbreviations, notes, etc.

53 53 Currently, in this system, we do not leverage handwriting recognition, as this method is flexible and supports a biologist s personal abbreviations, notes, etc. But we do plan to integrate recognition into a future version, where the recognition can help the biologist correct mistakes, or suggest better completions, etc. 53

54 ButterflyNet System Capture Structure Access Transform Share 54 And those are the tools and techniques that the ButterflyNet System contributes. 54

55 Prior Work a-book [Mackay, et al. 2002] paper notebook + PDA for lab biologists. Audio Notebook [Stifelman, et al. 2001] implicitly associate audio with notes. NISMap & others... [Cohen & McGee 2004] pen-and-paper are robust for the field. 55 ButterflyNet builds upon much prior work. While a detailed set is described in the paper, I will touch upon three important works that have heavily informed our designs. We were inspired by Mackay s work with a-book which combined a paper notebook + and a PDA to enable linking, and digital content viewing for laboratory biologists. This inspired our use of a handheld as the digital output to or pen-based input. Lisa Stifelman s AudioNotebook automatically associated notes from a lecture with the audio from the lecture. We used that idea as a basis for our automatic correlation between photos and notes. Finally, NISMap, RASA, and other systems from Cohen and McGee have demonstrated the robustness of pen and paper interfaces for field situations. We took this to heart in designing our field interactions. 55

56 Fieldwork studying field biologists Interaction Techniques mobile capture and access Evaluation first-use and longitudinal studies Current and Future Work integrated paper/digital interactions 56 56

57 A First-Use Study 2.5-hours each 14 Field Biologists 57 57

58 Experimental Design 10-min Tutorial Field Tools Lab Tools Questionnaires 58 [What we tested] [Just to see that it fit their work style, to validate our initial hypotheses, which were] 58

59 59 This is just a sample of all the data generated by our participants 59

60 Questionnaire Findings 60 [What we tested] [Just to see that it fit their work style, to validate our initial hypotheses, which were] [Questionnaire Findings] 60

61 Questionnaire Findings 61 [What we tested] [Just to see that it fit their work style, to validate our initial hypotheses, which were] [Questionnaire Findings] 61

62 Likes Dislikes Automatic Capture Bulky Pen, and Correlation adds to field time Requests Handwriting Recognition GPS Integration Support for audio & other content types 62 [Questionnaire Findings] [Open Questions] 62

63 Longitudinal Evaluation Used By: Field Biologists at JRBP/Stanford Anthropologists at Intel Digital Home Group Will Soon Be Used By: Cancer Researchers at Stanford NASA Researchers and more 63 We are currently evaluating ButterflyNet in long-term studies. Currently, ButterflyNet is being used by Biologists at Jasper Ridge and Stanford University, and also Anthropologists in Genevieve Bell s digital home group at Intel. Soon, we expect to deploy ButterflyNet with Cancer Researchers, and scientists working at NASA. And we have also been studying how ButterflyNet can benefit 63

Designers Poster #162 Technology for Design Education: A Case Study by Heidy Maldonado, Brian Lee, and Scott Klemmer 64 The design community, and design students, in particular.

64 Designers Poster #162 Technology for Design Education: A Case Study by Heidy Maldonado, Brian Lee, and Scott Klemmer 64 The design community, and design students, in particular. In this community, the correlation of photos with notes, and the ability to share design content, is very important. This work is lead by Brian Lee and Heidy Maldonado, and you can chat with them at their poster, #

65 Fieldwork studying field biologists Interaction Techniques mobile capture and access Evaluation first-use and longitudinal studies Current and Future Work integrated paper/digital interactions 65 Now, I will touch upon some of our current work, and explain a few of our future directions. 65

66 Map-Based Interaction Doctoral Consortium Poster #15 Physical-Digital Ensembles for Mobile Interaction by Ron B Yeh 66 Currently, we are exploring more interesting paper plus digital interactions, such as using a paper map to query for photos tagged with location information. The user circles a place on the paper map, and the resulting photographs are displayed on a nearby mobile device or large screen display. For more information and ideas on this, please see my doctoral consortium poster #15. 66

67 Future Work: Support More Content Types (e.g., Audio, Video, Sensor Data) Facets for Associating Content (e.g., Location, Tags, Weather, ) Collaboration Features (e.g., Flickr Photosharing) 67 In the future, we would like to support more types of content (such as audio, video, and sensor data). We would like to support more dimensions (or facets) for automatically associating content, such as GPS location, user supplied tags, and weather information. And, we would like to extend ButterflyNet s collaboration features to the web, by leveraging services such as the Flickr photosharing Website. 67

Summary Capture and Structure of Photos, Notes, and Specimens (Automatic

Broader Impact in Anthropology, Design, Archaeology, Medicine 68 In

and structure photos, notes, and specimens through both automatic and

It provides means to access, transform, and share research content,

68 Summary Capture and Structure of Photos, Notes, and Specimens (Automatic and Manual) Access, Transformation, and Sharing (Browser and Spreadsheet) Broader Impact in Anthropology, Design, Archaeology, Medicine 68 In summary, ButterflyNet contributes techniques to help biologists capture and structure photos, notes, and specimens through both automatic and manual techniques. It provides means to access, transform, and share research content, through the browser and multimedia spreadsheet. And, it had broader impact in domains such as anthropology, design, archaeology, and medicine 68

69 Summary Builds upon current practices. 69 But finally, the most important point, is that ButterflyNet builds upon current practices. If the digital pen fails in the field, it defaults to a regular inking pen and notebook. If the visual specimen tags don t work correctly, it becomes a regular coin envelope, that biologists use every day. 69

70 Acknowledgments Computer Science Terry Winograd; the members of BioACT Biology Rodolfo Dirzo, Nona Chiariello, Philippe Cohen, and more All the participants in our experiments Sponsors National Science Foundation Microsoft Research Nokia, Intel, and HP for technology donations 70 I would like to acknowledge Terry Winograd and the rest of our BioACT team, the biologists with whom we have worked, and our sponsors. ButterflyNet is open source, and is available on our homepage, at 70

stanford hci group and maryland hcil http://hci.stanford.edu/bio Questions? Ron B.

71 stanford hci group and maryland hcil Questions? Ron B. Yeh hci.stanford.edu/bio Thank you very much! 21:00 Minutes 71

72 72

73 Next Time Speech & Multimodal SpeechActs: A Spoken-Language Framework, Paul Martin, Frederick Crabbe, Stuart Adams, Eric Baatz, Nicole Yankelovich Multimodal Interfaces, Sharon Oviatt 73 73

ButterflyNet: A Mobile Capture and Access System for Field Biology Research

ButterflyNet: A Mobile Capture and Access System for Field Biology Research Ron B. Yeh 1, Chunyuan Liao 2, Scott R. Klemmer 1, François Guimbretière 2, Brian Lee 1, Boyko Kakaradov 1, Jeannie Stamberger