Assieme: Finding and Leveraging Implicit References in a Web Search Interface for Programmers

Size: px

Start display at page:

Download "Assieme: Finding and Leveraging Implicit References in a Web Search Interface for Programmers"

Darlene Hines
6 years ago
Views:

1 Assieme: Finding and Leveraging Implicit References in a Web Search Interface for Programmers Raphael Hoffmann, James Fogarty, Daniel S. Weld University of Washington, Seattle UW CSE Industrial Affiliates Meeting 2007

2 Programmers Use Search To identify an API To seek information about an API To find examples on how to use an API Example Task: Programmatically output an Acrobat PDF file in Java.

3 Example: General Web Search Interface

4 Example: Code-Specific Web Search Interface

5 Problems Information is dispersed: tutorials, API itself, documentation, pages with samples Difficult and time-consuming to locate required pieces, get an overview of alternatives, judge relevance and quality of results, understand dependencies. Many page visits required

6 With Assieme we Designed a new Web search interface Developed needed inference

7 Outline Motivation What Programmers Search For The Assieme Search Engine Inferring Implicit References Using Implicit References for Scoring Evaluation of Inference & User Study Discussion & Conclusion

8 Six Learning Barriers faced by Programmers (Ko et al. 04) Design barriers Selection barriers Coordination barriers combine? Use barriers What to do? What to use? How to How to use? Understanding barriers What is wrong? Information barriers How to

9 Examining Programmer Web Objective Queries See what programmers search for Dataset 15 million queries and click-through data Random sample of MSN queries in 05/06 Procedure Extract query sessions containing java 2,529 Manual looking at queries and defining regex filters Informal taxonomy of query sessions

10 Examining Programmer Web Queries

9 % Contain package, type or member name java SimpleDateFormat Use

11 Examining Programmer Web Queries 64.1 % 35.9 % Descriptive java JSP current date Selection barrier 17.9 % Contain package, type or member name java SimpleDateFormat Use barrier Contain terms like example, using, sample using currentdate code in jsp Coordination barrier

12 Assieme

13 Assieme packages/type s/members

14 Assieme relevance indicated by # uses

15 Assieme documentation

16 Assieme Filter search results

17 Assieme Summaries show referenced types

18 Assieme required libaries

19 Assieme example code

20 Assieme more info on types in sample code

21 Challenges How to put the right information? on the interface Get all programming-related data Interpret data and infer relationships

22 Outline Motivation What Programmers Search For The Assieme Search Engine Inferring Implicit References Using Implicit References for Scoring Evaluation of Inference & User Study Discussion & Conclusion

23 Assieme s Data Pages with code examples JAR files JavaDoc pages Queried Google on java ±import ±class Downloaded library files for all projects on Sun.com, Apache.org, Java.net, SourceForge.net Queried Google on overview-tree.html ~2,360,000 ~79,000 ~480,000 is crawled using existing search engines

24 The Assieme Search Engine infers 2 kinds of implicit references JAR files Pages with code examples Uses of packages, types and members Matches of packages, types and members JavaDoc pages?

25 Extracting Code Samples unclear segmentation code in a different language (C++) distracting terms in code line numbers

26 Extracting Code Samples remove HTML commands, but preserve line breaks remove some distracters by heuristics launch (error-tolerant) Java parser at every line break (separately parse for types, methods, and sequences <html> <head><title></title></hea d> A simple example: <body> A 1: import simple import java.util.*; example: 2: class class c { c { 1: import 3: HashMap m = java.util.*; m new = new 2: class HashMap(); c { 3: HashMap m 4: void = void f() { new f() m.clear(); { } } HashMap(); 4: 5: } } void f() { m.clear(); } 5: } back <a href= index.html >back</a

27 Resolving External Code References Naïve approach of finding term matches does not work: 1 import java.util.*; 2 class c { 3 HashMap m = new HashMap(); 4 void f() { m.clear(); } 5 } Reference java.util.hashmap.clear() on line 4 only detectable by considering several lines? Use compiler to identify unresolved names

28 Resolving External Code References Index packages/types/members in JAR files Compile & lookup compile unresolved names java.util.hashmap.clear() java.util.hashmap index lookup JAR files JAR files Utility function: # covered references (and JAR popularity) greedily pick best JARs put on classpath

29 Scoring Existing techniques Docs modeled as weighted term frequencies Hypertext link analysis (PageRank) do not work well for code, because: JAR files (binary code) provide no context Source code contains few relevant keywords Structure in code important for relevance

30 Using Implicit References to Improve Scoring Assieme exploits structure on Web pages and structure in code HTML hyperlinks code references

31 Scoring APIs Use text on doc pages and on pages with code samples that reference API (~ anchor text) Weight APIs by #incoming refs (~ PageRank) Web Pages Use fully qualified references (java.util.hashmap) and adjust term weights Filter pages by references Favor pages with accompanying text

32 Outline Motivation What Programmers Search For The Assieme Search Engine Inferring Implicit References Using Implicit References for Scoring Evaluation of Inference & User Study Discussion & Conclusion

33 Evaluating Code Extraction and Reference Resolution on 350 hand-labeled pages from Assieme s data Code Extraction Recall 96.9%, Precision 50.1% ( 76.7%) False positives: C, C#, JavaScript, PHP, FishEye/diff (After filtering pages without refs: precision Reference 76.7%) Resolution Recall 89.6%, Precision 86.5% False positives: Fisheye and diff pages False negatives: incomplete code samples

34 User Study Assieme vs. Google vs. Google Code Search Design 40 search tasks based on queries in logs: query socket java Write a basic server that communicates using Sockets Find code samples (and required libraries) 4 blocks of 10 tasks: 1 for training + 1 per interface Participants 9 (under-)graduate students in Computer Science

35 User Study Solution Quality 0 seriously flawed.5 generally good but fell short in critical regard 1 fairly complete 1.0 F(1,258)=55.5 p <.0001 F(1,258)=6.29 p.013 * * quality (± SEM) Assieme Google GCS

36 User Study # Queries Issued #queries (± SEM) F(1,259)=9.77 p.002 F(1,259)=6.85 p.001 * * 0.0 Assieme Google GCS

37 User Study Task Time F(1,258)=5.74 p.017 * significant 150 F(1,258)=1.91 p.17 seconds (± SEM) Assieme Google GCS

38 Outline Motivation What Programmers Search For The Assieme Search Engine Inferring Implicit References Using Implicit References for Scoring Evaluation of Inference & User Study Discussion & Conclusion

39 Discussion & Conclusion Assieme a novel web search interface Programmers obtain better solutions, using fewer queries, in the same amount of time Using Google subjects visited 3.3 pages/task, using Assieme only 0.27 pages, but 4.3 previews Ability to quickly view code samples changed participants strategies

Thank You Raphael Hoffmann Computer Science & Engineering University of Washington raphaelh@cs.washington.

Weld Computer Science & Engineering University of Washington weld@cs.washington.

40 Thank You Raphael Hoffmann Computer Science & Engineering University of Washington James Fogarty Computer Science & Engineering University of Washington Daniel S. Weld Computer Science & Engineering University of Washington This material is based upon work supported by the National Science Foundation under grant IIS , by the Office of Naval Research under grant N , SRI International under CALO grant and the Washington Research Foundation / TJ Cable Professorship.

Assieme: Finding and Leveraging Implicit References in a Web Search Interface for Programmers

Assieme: Finding and Leveraging Implicit References in a Web Search Interface for Programmers Raphael Hoffmann, James Fogarty, Daniel S. Weld Computer Science & Engineering University of Washington Seattle,