Pliny and Fixr Meeting. September 15, 2014

Transcription

1 Pliny and Fixr Meeting September 15, 2014

2 Fixr: Mining and Understanding Bug Fixes for App-Framework Protocol Defects (TA2) University of Colorado Boulder September 15, 2014

3 Fixr: Mining and Understanding Bug Fixes for App-Framework Protocol Defects (TA2) Bor-Yuh Evan Chang Ken Anderson Pavol Cerny Sriram Sankaranarayanan Tom Yeh University of Colorado Boulder University of Colorado Boulder September 15, 2014

4 A bug that manifests spectacularly

5 A bug that manifests spectacularly

6 A bug that manifests spectacularly

7 A bug that manifests spectacularly Crash

8 A bug that manifests spectacularly Crash caused by an app-created memory leak

9 Framework Dev s say...

10 Framework Dev s say...

11 Framework Dev s say... Do not keep long-lived references to a context-activity

12 Framework Dev s say... Do not keep long-lived references to a context-activity I don t know how I created a long-lived reference to an Activity!

13 Framework Dev s say... Do not keep long-lived references to a context-activity I don t know how I created a long-lived reference to an Activity! Often: A misunderstanding of a library causes the library to keep the Activity reference.

14 Framework Dev s say... Do not keep long-lived references to a context-activity Bug from violating I don t know how I created a long-lived reference to an Activity! Often: A misunderstanding of a library causes the library to keep (implicit) framework protocol rules the Activity reference.

15 Imagining a post-muse scenario... I don t know how I created a long-lived reference to an Activity! for xxxxxxxxxxxx

16 Elsewhere, following the state of practice for debugging leaks...

17 Elsewhere, following the state of practice for debugging leaks Run the app

18 Elsewhere, following the state of practice for debugging leaks Run the app 2. Watch the heap usage

19 Elsewhere, following the state of practice for debugging leaks Run the app 2. Watch the heap usage 3. Dump the heap. Dig around and finally find the culprit!

20 Elsewhere, following the state of practice for debugging leaks Run the app 2. Watch the heap usage 3. Dump the heap. Dig around and finally find the culprit!

21 Elsewhere, following the state of practice for debugging leaks Run the app 2. Watch the heap usage 3. Dump the heap. Dig around and finally find the culprit! 4. Commit a bugfix

22 Elsewhere, following the state of practice for debugging leaks Run the app 2. Watch the heap usage 3. Dump the heap. Dig around and finally find the culprit! 4. Commit a bugfix 5. Bugfix is picked up by Fixr Fixr

23 A Fixr-enabled IDE responds... I don t know how I created a long-lived reference to an Activity!

24 A Fixr-enabled IDE responds... I don t know how I created a long-lived reference to an Activity!

25 A Fixr-enabled IDE responds... It looks like you ve created a memory leak like and 100,000 others. Would you like to apply? I don t know how I created a long-lived reference to an Activity!

26 A Fixr-enabled IDE responds... It looks like you ve created a memory leak like and 100,000 others. Would you like to apply? the bugfix is transferred I don t know how I created a long-lived reference to an Activity!

27 Summary: Mine specifications of framework rules (indirectly) from bugfixes Leverage volume and variety of bugfixes made by the crowd of client app developers

28 Summary: Mine specifications of framework rules (indirectly) from bugfixes Leverage volume and variety of bugfixes made by the crowd of client app developers

29 Summary: Mine specifications of framework rules (indirectly) from bugfixes Leverage volume and variety of bugfixes made by the crowd of client app developers toolify stackoverflow

30 Fixr: Proposed System MUSE

31 Fixr: Proposed System fix MUSE

32 Fixr: Proposed System E.g., Two successive versions of source code fix MUSE

33 Fixr: Proposed System fix MUSE

34 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications fix MUSE

35 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications semantic delta fix MUSE

36 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications E.g., Diff in relevant flow-insensitive summary semantic delta fix MUSE

37 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications semantic delta fix MUSE

38 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification semantic delta fix MUSE

39 Fixr: Proposed System E.g., framework Deltar: Inferring Semantic Deltas and Repair Specifications semantic delta invariant + app pre for bug + fix repair specification fix MUSE

40 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification semantic delta fix MUSE

41 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE

42 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification

43 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications fix semantic delta MUSE probabilistic repair specification E.g., generalized repair spec with confidence measure

44 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification

45 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification Patchr: Detecting Potential Bugs and Synthesizing Patches

46 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification patch Patchr: Detecting Potential Bugs and Synthesizing Patches

47 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification patch E.g., bug evidence and patch Patchr: Detecting Potential Bugs and Synthesizing Patches

48 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification patch Patchr: Detecting Potential Bugs and Synthesizing Patches

49 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification Harvestr: Social Validation and Mining of Fixes patch Patchr: Detecting Potential Bugs and Synthesizing Patches

50 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification social delta Harvestr: Social Validation and Mining of Fixes patch Patchr: Detecting Potential Bugs and Synthesizing Patches

51 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification E.g., bug fix confirmation social delta Harvestr: Social Validation and Mining of Fixes patch Patchr: Detecting Potential Bugs and Synthesizing Patches

52 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification interaction E.g., bug fix confirmation Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches

53 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification interaction Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches

54 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification interaction Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches

55 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code

56 Fixr: Proposed System Deltar: Inferring Semantic Deltas and Repair Specifications repair specification Urepair: Deriving Probabilistic Repair Specifications semantic delta fix MUSE probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code

57 Fixr: Proposed System semantic statistical-semantic syntactic social Deltar: Inferring Semantic Deltas and Repair Specifications fix semantic delta repair specification MUSE Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code

58 Fixr: Proposed System semantic statistical-semantic syntactic social Deltar: Inferring Semantic Deltas and Repair Specifications fix semantic delta repair specification MUSE Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification interaction Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Goal: Create a positive feedback loop commit Patches to Code derive high-quality repair specifications

59 semantic statistical-semantic syntactic social Deltar: Inferring Semantic Deltas and Repair Specifications fix semantic delta repair specification MUSE Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code

60 symbolic program analysis Bor-Yuh Evan Chang Team Sriram Sankaranarayanan numerical-probabilistic program analysis semantic statistical-semantic syntactic social Deltar: Inferring Semantic Deltas and Repair Specifications software engineering for big data fix semantic delta repair specification MUSE Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes Ken Anderson social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code user-centered big data analytics program synthesis Tom Yeh Pavol Cerny

61 Evaluation Questions

62 Evaluation Questions Iterative and incremental design and evaluation of the Fixr loop

63 Evaluation Questions Iterative and incremental design and evaluation of the Fixr loop Effectiveness of Bugfix Transfer: Given an isolated bugfix, can we derive high-quality repair specifications to lead to useful patches?

64 Evaluation Questions Iterative and incremental design and evaluation of the Fixr loop Effectiveness of Bugfix Transfer: Given an isolated bugfix, can we derive high-quality repair specifications to lead to useful patches? Effectiveness of Bugfix Seeding: Can we isolate likely bugfixes from source repositories?

65 Bor-Yuh Evan Chang Sriram Sankaranarayanan semantic statistical-semantic syntactic social Deltar: Inferring Semantic Deltas and Repair Specifications fix semantic delta repair specification MUSE Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification interaction commit Harvestr: Social Validation and Mining of Fixes Ken Anderson social delta patch Patchr: Detecting Potential Bugs and Synthesizing Patches Code Tom Yeh Pavol Cerny

66 Deltar Deltar: Inferring Semantic Deltas and Repair Specifications repair specification semantic delta fix

67 Bug: On Android <4 aview.settag(..., anobject)

68 Bug: On Android <4 aview.settag(..., anobject) if anobject can reach aview

69 Bug: On Android <4 aview.settag(..., anobject) if anobject can reach aview Goal: Produce the above ( bug pre ) with framework invariant and fix

70 Subtask 1.1: Summarizing App Commits

71 Subtask 1.1: Summarizing App Commits Diff coarse semantic summaries E.g., points-to graphs abstracted to appframework crossings

72 Subtask 1.1: Summarizing App Commits Diff coarse semantic summaries E.g., points-to graphs abstracted to appframework crossings Refine diffs

73 Subtask 1.2: Approximating Framework Properties

74 Subtask 1.2: Approximating Framework Properties Need 1: Scalable verification of framework invariants (on the bugfixed version) Fissile Types: intertwined invariant-based and operational-based reasoning ( almost everywhere dependent-refinement types)

75 Subtask 1.2: Approximating Framework Properties Need 1: Scalable verification of framework invariants (on the bugfixed version) Fissile Types: intertwined invariant-based and operational-based reasoning ( almost everywhere dependent-refinement types) Need 2: Ming framework invariants Refine semantic diff to a framework-only specification

76 Urepair repair specification Urepair: Deriving Probabilistic Repair Specifications probabilistic repair specification

77 Traditional Program Analysis Program (Fragment) Program Analyzer Annotations

78 Traditional Program Analysis Program (Fragment) Program Analyzer Annotations Program Database

79 Connection with Repair Specifications Repair Spec #1 Repair Spec #2 Probabilistic Repair Synthesis Repair Spec #N Goal: Synthesize multiple repair specifications into a likely candidates.

80 Bayesian Reasoning Prior Hypothesis Bayesian Update Posterior Hypothesis Observational Data

81 Bayesian Program Analysis class GadgetStore{ List<Gadget> lst; void addtolist( Gadget x ){ // Can you guess what the function does? pre (x!= null); } } post( len(x_0) <= len(x_p) <= len(x_0) +1 );

82 Approach There are lots of soft information sources that program analysis tools can use. However, we do not use them Fear of unsoundness. Integrate multiple sources into prior annotations. Allow weights for prior facts to signify degree of belief.

83 Prior Beliefs class GadgetStore{ List<Gadget> lst; void addtolist( Gadget x ){ // Can you guess what the function does? pre (x!= null); Prior Weight: 0.9 } } post( len(x_0) <= len(x_p) <= len(x_0) +1 );

84 Program Analysis: Bayesian Updating Program analysis is operates over distributions of static analysis facts. Assume a level of belief in the correctness of a function. Pr(Assertion FunctionCorrect) = Pr(FunctionCorrect Assertion) Pr(Assertion) Pr(FunctionCorrect)

85 Current Progress Ongoing investigations into probabilistic program analysis. Sankaranarayanan et al. PLDI 2013 Chakarov et al. CAV 2013, SAS Goals: Support Bayesian interpretation. Design program analysis tools for Bayesian Update. Integrate into repair specification synthesis.

86 Patchr probabilistic repair specification patch Patchr: Detecting Potential Bugs and Synthesizing Patches

87 Patchr How do we validate repair specifications? We synthesize appropriate patches [Patchr]. and validate the patches using human input (mturk, pull requests) [part of Harvestr] Motivation for developers to provide feedback: 1. Patches easier to understand than repair specs 2. Patches, if correct, immediately useful to developers

88 Subtask 3.1: Applying and concretizing repair specifications Subtask 3.2: Synthesis using MUSE queries

89 Subtask 3.1: Applying and concretizing repair specifications 1. Finding where to apply a repair spec 2. Finding how to apply it (concretizing repair specs): Example: suppose repair spec says always call methods A and B before C suppose the program-under-repair already calls A patch: call B after A and before C

90 Subtask 3.2: Synthesis using MUSE queries While producing patches, we need to understand the program-under-repair Develop techniques for synthesis with rich queries: pre- and post-conditions of relevant methods invariants Studied for finite-state systems as synthesis from libraries components

91 Team: Pavol Cerny Vaibhav Singh (MS) Currently recruiting PhD students and postdocs

92 Harvestr fix interaction Harvestr: Social Validation and Mining of Fixes social delta patch

93 Harvestr Runtime Input: A possible bug fix. Output: Yes! It is a bug fix. (or not) Training: Input: Lots and lots of apps and repos Output: Social proof

94 Social Validation Scenario 1: Developer releases an app. Users leave comments about a bug. Developer releases an update containing a bug fix. Users no longer leave comments about the bug.

95 Social Validation Scenario 2: Developer releases an app. After 1 day Developer releases an update.

96 Social Validation Scenario 3: Developer releases an app. After 30 days. Developer releases an update.

97 Social Validation Scenario 4: Developer A releases an update. After 1 day Developer B releases an update. Two updates are similar.

98 Big data problem Volume Variety Velocity Veracity

99 Volume Number of apps (1 millions) x Number of versions (10) x Number of source files (100) Current: 200K apps, 2-7 versions/app.

100 Velocity New app (10/day) New update (100/day) New review (1000/day) New repo (1/day) New commit (1/repo/day)

101 Variety user ratings download counts version increments update intervals app s description user reviews ( many more)

102 Veracity Biases Low-level vs. high-level Uncertainties Precision vs. recall Abnormality Data entry, Outliers

103 Fixr Big Code Architecture MUSE commit Code

104 Scaling Fixr Up Scalability requirements to make Fixr operational Target: Collect at least 50% of public Android apps on Google Play and Github > 500K apps => ~100M files Handle updates to revision histories 10 updates on Google Play; 50 commits on Github Velocity: 10 new apps, 100 updates, 1000 user reviews per day; 1 new repo + 1 new commit per day

105 Data Profile Data is Text: Source code, user reviews, bug reports Binary: git repos, application assets, Meta: ratings, download counts, version numbers More importantly, Data is read-only highly interconnected

106 Proposed Architecture MUSE Front End Application Layer DataStax Enterprise Pig Hadoop Solr Redis Service Layer Cassandra Storage Layer

107 Discussion (I) Cassandra used to store all read-only data very fast at streaming results of key-based queries Solr used to index the important metadata and to perform complex queries Pig/Hadoop used to process applications in batch Redis used to cache results of frequent queries

108 Discussion (II) The artifacts generated by Deltar, Urepair, Patchr, and Harvestr also stored in Cassandra and tagged by type Allows Hadoop jobs to find them and process them at scale Not clear, currently, whether we will incorporate a scalable graph technology such as Titan, MapD, or Neo4j May be okay initially to handle interconnections between artifacts via row-key references

109 Scaling App/Repo Collection Have scripts to download Android apps and their associated artifacts from Google Play and Github Need to scale the capabilities of these scripts by one order of magnitude Need to automate the collection to happen on daily basis Will explore the use of Apache Nutch for this purpose Enables massively parallel web crawling Each crawling thread can be engineered to store retrieved assets in Cassandra

110 Computing Infrastructure Currently have access to a cluster of 7 machines 4 dedicated Cassandra nodes 1 analytics machine with 128 GB of memory 1 compute server used to host VMs 1 storage server used to perform backups Acquired 5 new servers last week Goal is to combine all 12 machines into a single OpenStack cluster and then provision servers as needed