Configuring Topic Models for Software Engineering Tasks in TraceLab

Transcription

1 Configuring Topic Models for Software Engineering Tasks in TraceLab Bogdan Dit Annibale Panichella Evan Moritz Rocco Oliveto Massimiliano Di Penta Denys Poshyvanyk Andrea De Lucia TEFSE 13 San Francisco, CA

2 Textual Information Information Retrieval Techniques Maintenance Tasks

3 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration

4 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration imported from natural language community ad-hoc configuration Assumption: source code has the same characteristics as natural language

5 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration

6 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration Sub-optimal results

7 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration [Hindle et ICSE 12]: source code is more regular and predictable than natural language Sub-optimal results

8 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration ICSE 2013 Sub-optimal ICSE 2013

9 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration ICSE 2013 Sub-optimal results

10 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration ICSE 2013 Sub-optimal results Near-optimal results

11 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration ICSE 2013 Sub-optimal results Near-optimal results

12 Textual Information Information Retrieval Techniques Maintenance Tasks LDA Require Parameter Calibration Traceability ICSE 2013 Sub-optimal results Near-optimal results

13 What is LDA?

14 Latent Dirichlet Allocation (LDA) Topic model that generates the distribution of latent topics from textual documents

15 Latent Dirichlet Allocation (LDA) Topic model that generates the distribution of latent topics from textual documents LDA logging http connect save GUI export

16 Source Artifacts Target Artifacts Preprocessing Term-by-Document Matrix

17 Source Artifacts Target Artifacts Preprocessing Term-by-Document Matrix LDA

18 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA

19 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Topic by Documents

20 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Topic by Documents Probability that document is related to topic

21 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Topic by Documents

22 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Topic by Documents

23 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Compute document to document similarity Topic by Documents Compute query to document similarity Cluster documents by topics

24 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Topic by Documents

25 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Words to Topic Topic by Documents

26 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Words to Topic Topic by Documents

27 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Words to Topic Topic by Documents Generate topic labels (top 5 words)

28 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA

29 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Configuration # of iterations # of topics α β

30 Source Artifacts Target Artifacts Preprocessing Input Parameters Term-by-Document Matrix LDA Configuration # of iterations # of topics α β Number of Gibbs samplings of LDA model

31 Number of topics Low High

32 Number of topics Low High General topics

33 Number of topics Low High General topics Specific topics

34 α, influences the smoothness of documents to topics distribution Low α High α

35 α, influences the smoothness of documents to topics distribution Low α High α

36 α, influences the smoothness of documents to topics distribution Low α High α

37 α, influences the smoothness of documents to topics distribution Low α High α

38 β, influences the smoothness of topics to words distribution Low β High β

39 β, influences the smoothness of topics to words distribution Low β High β

40 LDA parameters significantly 1,000 different configurations of LDA parameters influence the results Evaluate the Average Precision on EasyClinic

41 LDA parameters significantly 1,000 different configurations of LDA parameters influence the results Evaluate the Average Precision on EasyClinic

42 LDA parameters significantly 1,000 different configurations of LDA parameters influence the results Few configurations produce good results Evaluate the Average Precision on EasyClinic High variability in results LDA-GA

43 LDA GA: automatically calibrate the input parameters of LDA using genetic algorithms

44 LDA GA: automatically calibrate the input parameters of LDA using genetic algorithms Dataset LDA-GA

45 LDA GA: automatically calibrate the input parameters of LDA using genetic algorithms LDA parameters Dataset LDA-GA # of iterations # of topics α β LDA

46 LDA GA: automatically calibrate the input parameters of LDA using genetic algorithms LDA parameters Dataset LDA-GA # of iterations # of topics α β LDA Oracle Traceability Results

47 LDA GA: automatically calibrate the input parameters of LDA using genetic algorithms LDA parameters Dataset LDA-GA # of iterations # of topics α β LDA Dataset dependent Oracle, Task independent Oracle Traceability Results

48 Evaluating the LDA parameter configuration

49 Topic by Documents

50 Topic by Documents Dominant Topics

51 Topic by Documents LDA Model Dominant Topics

52 Topic by Documents LDA Model Dominant Topics

53 Topic by Documents LDA Model Dominant Topics

54 Topic by Documents LDA Model

55 Topic by Documents LDA Model Cohesion (similarity): how related the documents in the same clusters are Separation (dissimilarity): how distinct a cluster is from other clusters

56 Topic by Documents LDA Model Silhouette coefficient

57

58 Cohesive Separated Documents are rearranged more closely in the n- dimensional space

59 We propose a genetic algorithm to find those solutions

60 Choose a set of Random LDA parameters

61 Choose a set of Random LDA parameters

62 Choose a set of Random LDA parameters LDA Determine cohesion and separation

63 Choose a set of Random LDA parameters LDA Determine cohesion and separation Cohesion and Separation (Silhouette)

64 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters

65 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters

66 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

67 Choose a set of Random LDA parameters Crossover LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

68 Choose a set of Random LDA parameters Crossover LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

69 Choose a set of Random LDA parameters Crossover LDA Determine cohesion and separation Select best LDA parameters Mutation Crossover, mutation

70 Choose a set of Random LDA parameters Crossover LDA Determine cohesion and separation Select best LDA parameters Mutation Crossover, mutation

71 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

72 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

73 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters Crossover, mutation

74 Choose a set of Random LDA parameters LDA Determine cohesion and separation Select best LDA parameters Return best LDA parameters # of iterations # of topics α β Crossover, mutation

75 Precision/Recall EasyClinic

76 Precision/Recall EasyClinic LDA-GA Exhaustive (best of 1,000 configurations) ad-hoc configuration [ICPC 10]

77 LDA GA in TraceLab R Implementation of LDA-GA

78 LDA GA in TraceLab Genetic algorithm settings LDA-GA settings

79

80 Textual Information Information Retrieval Techniques Maintenance Tasks Require Parameter Calibration ICSE 2013 Sub-optimal results Near-optimal results

81 Textual Information Information Retrieval Techniques Maintenance Tasks Other IR techniques: (LSI, RTM) Require Parameter Calibration Other parameters: (preprocessing) ICSE 2013 Sub-optimal results Near-optimal results

82 Thank you! Questions? GA/