A Survey of Bug Tracking Tools: Presentation, Analysis and Trends

Transcription

1 A Survey of Bug Tracking Tools: Presentation, Analysis and Trends Trajkov Marko, Smiljkovic Aleksandar Department of Computer Science, University of Belgrade Abstract Nowadays, when project are so extensive, the one sure thing that will happen is bag. Because of that, it becomes very important to have appropriate bug tracking system. Poorly designed bug tracking systems can are sometimes blamed for bad information exchanging between developers. The purpose of this paper is to present some most common used such systems, to address his problems and give some direction for enhancement. 1. Introduction In the introductory section we specify the following three issues of importance: (a) Definition of the general field of interest for this research, (b) Definition of the special terminology of interest for this research, and (c) Definition of the viewpoint of interest for this research (definition of the axiomatic requirements of this research). Today, the using bug tracking system for tracking bugs and other issues is well spread. Bug tracking systems are using for organizing and monitoring bugs. Without this systems, monitoring a large amount of bugs will be impossible or very hard. Because of that, today you can find a lot of this bug tracking systems. They very differ by quality, security, costs, and functionality they offer to the user. We can observe bug tracking system as a part of much wider system for project tracking or as an separate system. Both cases have the advantages and disadvantages but form this point of view we will observe them as an separate systems. The reason for this is to reduce the scope of research and therefore easier and more precise treat the problem. From the other side, the project tracking systems are more tighter related to the specific problem. Previous research has shown that items such as stack traces, steps to reproduce, observed and expected behavior, test cases, and screenshots are often omitted by reporters [4, 6] and that items are preferred information for developers. The effect of this delay is that bugs take longer to be fixed and more and more unresolved bugs accumulate in the project s bug tracking system. The audience for this paper we can split into two groups. The first of them are individuals or companies who work on projects which require some bug or issue tracking system. This paper should help them to

2 better understand the ground and chose appropriate tool for their system. The second group is developers of the same. They should know the competition. Also this paper should help them to build the system which can be better than others in the set of features and functionality, and the system which can satisfy the larger population. In the next of this document we present issues of importance for the understanding this paper (Section 2). Further, we present existing solutions of the problem and their criticism (Section 3). The paper closes with trends and the optimal solutions for future bug tracking systems (Section 4) followed by conclusions (Section 5). 2. Problem statement In this section we focus on the following issues of importance for the understanding of the basic orientation of this research: (a) Problem definition, (b) Elaboration of problem importance (why is the problem important?), and (c) An assessment of the problem development trends (why will the importance of this problem grow over time?). Bug Tracking Systems allow individual or groups of developers to keep track of outstanding bugs in their product effectively. Having complete information in the initial bug report helps developers to quickly resolve the bug. So choosing a good bug tracking system for your product helps you to reduce downtime, increase productivity, raise customer satisfaction, and improve communication between developers. Nowadays, modern projects become much more complex, robust and harder to keep it track with them. Parallel with that, it is becoming much more harder to keep in tracks a huge amount of bugs. On the other side, the projects passes through many phases such as alpha, beta, release candidate and through many versions since first version. Because of that it is important to keep in tracks of all malfunctions which are noticed in projects life circle. The complexity of project will grow over time. The versions of existing projects will continue to increase. The same bugs will continue to appear in every generation of developers. So, it is recommended to keep in record all of this malfunctions to improve future software development. 3. Existing solutions of the problem and their criticism In this section, we give a short overview of existing solutions of the problem specified in the Problem Definition section, and we criticize the existing solutions from the point of their usability. In general, the overviewed solutions give excellent results under the specific problem of interest, but they do not address the general problem of interest for this research. This section starts with an overview for each of selected solutions, gives the following main points: (a) The basic information of each solution, (b) Specific details for each selected solution, (c) Further

3 development trends of the approach, and (d) A criticism of the solution, and finally (e) Possible improvements that could overcome the noticed drawbacks. It concludes with a classification of each elaborated solution. The classification criteria were chosen to reflect the essence of the basic viewpoint of this research. It will summarize all important parts of elaborated solutions Presentation of existing solutions and their drawbacks This section is divided in several subsections, one per each solution. For each solution, several sentences per paragraph are given Bugzilla Bugzilla very popular, actively maintained and free bug tracking system, used and developed together with Mozilla, giving it considerable credibility. Bugzilla is based on Perl and once it is set up, it seems to make its users pretty happy. It's not highly customizable, but in a odd way, that may be one of its features: Bugzilla installations tend to look pretty much the same wherever they are found, which means many developers are already accustomed to its interface and will feel they are in familiar territory. Bugzilla has a system that will send you, another user, or a group that you specify the results of a particular search on a schedule that you specify. Bugzilla has a very advanced reporting systems and you can create different types of charts including line graph, bar graph or pie chart Mantis Mantis is a free web-based bug tracking system. It is written in the PHP scripting language and works with MySQL, MS SQL, and PostgreSQL databases and a webserver. Mantis can be installed on Windows, Linux, Mac OS and OS/2. Almost any web browser should be able to function as a client. It is released under the terms of the GNU General Public License (GPL). The main complaint is its interface which doesn t meet modern standards. On the other hand, is easy to navigate, even for inexperienced users. There not exists some advanced features such as charts and reports. In short, the whole system is sloppily done, there are plenty of bugs and very little functionality BugTracker.NET BugTracker.NET is a free, open-source, web-based bug tracker or customer support issue tracker written using ASP.NET, C#, and Microsoft SQL Server Express. BugTracker.NET is easy to install and learn how to use. When you first install it, it is very simple to setup and you can start using it right away. Later, you can change its configuration to handle your needs. It has a very intuitive interface for generating lists of bugs. It has two very useful features. First of them is a screen capture utility that enables you to capture the screen, add annotations and post it as bug in just a few clicks. The second feature is the fact that it can integrate with your Subversion repository so that you can associate file revision check ins with bugs Flyspray Flyspray is a web-based bug tracking system written in PHP. Flyspray is free software, released under the General Public License. This essentially means that you can get Flyspray and use it free of charge. The source code is available, and everyone are welcome to modify it to suit their needs. Its web pages describe it as "uncomplicated", and the list of features includes: multiple database support (currently

4 MySQL and PGSQL), multiple projects, 'watching' tasks, with notification of changes (via or Jabber), comprehensive task history, CSS theming, file attachments, advanced search features, RSS/Atom feeds, wiki and plaintext input, voting, dependency graphs Redmine Redmine is a flexible web-based project management web application. Written using Ruby on Rails framework, it is cross-platform and cross-database. Redmine is open source and released under the terms of the GNU General Public License. Redmine is flexible issue tracking system. You can define your own statuses and issue types. He support multiple projects and subprojects. Each user can have a different role on each project. Interface is very simple, intuitive and easy to navigate. Shortly, this is very good product and our recommendation Bug-Track Bug-Track is web-based defect and bug tracking software allows you to document, manage and assign all of your bugs and tasks and empowers you to organize your bugs, defects or issues into distinct projects. It can run on virtually any web-server like Microsoft, Linux, Unix, etc... Since it is an commercial application it is expected that it is better than other free products. But it isn t true. He has nothing new and better than other free bug tracking systems. One better thing is fact that he have more intuitive interface then others and that is his only benefit Bugzero Bugzero is a web-based bug, defect, issue and incident tracking software. Its single code base supports both Windows and Unix (based on Java ) and supports database systems including Access, MySQL, SQL Server, Oracle, and etc. Bugzero can be customized for software bug tracking, hardware defect tracking, and help desk customer support issue and incident tracking. Bugzero have intuitive interface but he lacks form features. The main drawback is the fact that Bugzero is an commercial product and you can find much better product for free Conclusion about existing solutions From all above presented, we conclude that, among the existing solutions, the none of them can be treated as the best one, for the general solution of this research. Every of them have some advantages and disadvantages. Some of them have some feature more than others but in the general, the set of features are the same Classification criteria and the classification tree For classification criteria for this research are selected the most common features that mosses most of the selected bug tracking tools. This features are useful for an average user and they give an easy to use bug tracking system. Decision making using this criteria is not good and some more advanced criteria will be given in the next section (Section 4). The classification criteria of interest for this research, are given in Table 1. All selected classification criteria are explained in the caption of Table 1, and elaborated in the paragraph to follow. Such that criteria are widely used in various systems that are in use today.

5 Search notifications Reports Charts Time Tracking RSS/Atom Feed Configurable Free Bugzilla yes yes yes yes yes no no yes Mantis yes no no no yes yes no yes BugTracker.NET yes yes yes yes yes no yes yes Flyspray yes yes yes no no yes yes yes Redmine yes yes yes yes yes yes no yes Bug-Track yes yes yes yes no yes no no Bugzero yes yes yes no no no no no Table 1: Classification criteria. Legend: search, notifications, reports, charts, time tracking, RSS/Atom Feed, Configurable, Free. Explanation: This table summarize criteria that are used in decision making process of choosing suitable bug tracking system. Table 1. summarize criteria that are used in decision making process of choosing suitable bug tracking system. That criteria are: search, notifications, reports, charts, time tracking, RSS/Atom Feed, configurable (can system be configure its settings to meet client standards). The brief description of each criteria are given next. The classification tree, derived from the above introduced classification criteria, is presented in Figure 1. Each leaf of the classification tree is given a name, as described in the caption of Figure 1. For each leaf, the list of existing solutions is given. Figure 1: Classification tree. Legend: Charts, Configurable, Free. Explanation: This tree summarize basic criteria that are used in decision making process of choosing suitable bug tracking system. Now, we can summarize all criteria given above in Table 1 and Figure 1. Search is very useful criteria and it is present in all selected products. notifications gives user opportunity to be noticed about happenings in the current bug tracking system. The user do not need to check frequently bug tracking

6 system for new changes. All he need is an account. Reports give user a brief and concise overview about past happenings in our system. Charts give clear graphical view of selected criteria which is very intuitive to the human being. Time tracking is an feature that give information about happenings of some specific bug trough time. Like an RSS/Atom feed gives user opportunity to be noticed about happenings in the current bug tracking system. Configurable system is capable to be configured to meet certain user needs, so the system should be configurable as much as possible, because that will help to satisfy the larger population of customers. At the end, the much important issue for choosing the right bug tracking system is the fact is it free or not and how much he cost. 4. Trend and the optimal solution for future When a user submits a bug report she is asked many questions: What is the name of the product? In which plugin/component? What is the Build ID? What is the bug about? What are the steps to reproduce the bug? Any additional information? However, the initial information provided in a bug report is often incomplete and developers often have follow-up questions: Do you have flash installed? Is there any screenshot? Getting replies by users takes time (often weeks) and slows down the progress on open bugs. We therefore propose to build expert systems that automatically ask relevant questions and gather all required information once an initial failure description has been provided. The selection and order of questions should not be static (as in current bug tracking systems), but rather depend on previous responses. We believe that such an expert system can provide better bug descriptions. The classification criteria of interest for this research, are given in Table 2. All selected classification criteria are explained in the caption of Table 1, and elaborated in the paragraph to follow. Stack Steps to Observed Expected Test Screenshots Dependencies Trace Reproduce behavior Behavior Cases Bugzilla no no no no no no no Mantis no yes no no no no no BugTracker.NET no no no no no yes yes Flyspray no no no no no no no Redmine no no no no no no no Bug-Track no no no no no no no Bugzero no no no no yes no yes Table 2: The better selection criteria for proposed approach. Legend: stack traces, steps to reproduce, observed behavior, expected behavior, test cases, screenshots and dependencies. Explanations: This table showing that all of today bug tracking tools lacks from some important properties. The essence of this proposed system is presented in better bug description. As you can see form the Table 2, the most of the selected criteria are not supported in current bug tracking systems. The current bug tracking systems rely on some different selection criteria, which sometimes cannot give satisfying results. So, we believe that including selection criteria like stack traces, steps to reproduce, observed behavior, expected behavior, test cases, screenshots and dependencies will improve current bug tracking systems.

7 The advantages of the improved set of criteria will give an user better information about current bug. This will lead to faster locating of the current defect and to the faster respond to it. 5. Conclusion Current bug tracking systems do not effectively collect all of the information needed by developers. Without this information developers cannot resolve bugs in a timely fashion and so we believe that improvements to the way issue tracking systems collect information are needed. We summarized criteria that are used in modern bug tracking systems. Such criteria often doesn t give appropriate results in describing bug. So, we proposed an improved set of criteria that will give much more satisfying solution for the current system. This work can be important to the designers of the future bug and defect tracking systems. They should know importance of selected criteria for describing bug because a well described bug will be easier to be find and solved. Designing such an system will give us an answer to our assumption. That will confirm our work and apply new ideas to the current bug tracking systems. At the end we can notice that current bug tracking system are web oriented, and such that trend will be continued. It is very important to design system that selection criteria for bug describing will meet such trend. 6. References (annotated bibliography) [1] J. Anvik, L. Hiew, and G. C. Murphy. Who should fix this bug?, Proceedings of the ICSE-2006, 28 th International Conference on Software engineering, 2006, pp [2] J. Aranda and G. Venolia. The secret life of bugs: Going past the errors and omissions in software repositories, Proceedings of the ICSE -2009, Proceedings of the 31st International Conference on Software engineering (to appear), [3] S. Artzi, S. Kim, and M. D. Ernst. Recrash: Making software failures reproducible by preserving object states, In Proceedings of the 22nd European Object-Oriented Programming Conference, 2008, pp [4] N. Bettenburg, S. Just, A. Schroter, C. Weiss, R. Premraj, and T. Zimmermann. What makes a good bug report?, Proceedings of the FSE-2008, 16th International Symposium on Foundations of Software Engineering, November 2008, pp [5] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. Duplicate bug reports considered harmful... really?, Proceedings of the ICSM-2008: Proceedings of the 24th IEEE International Conference on Software Maintenance, 2008, pp [6] S. Breu, J. Sillito, R. Premraj, and T. Zimmermann. What developers ask about your bug?, Technical report, University of Calgary, December [7] P. Fritzson, T. Gyimothy, M. Kamkar, and N. Shahmehri. Generalized algorithmic debugging and testing, Proceedings of the PLDI-1991, ACM SIGPLAN Conference on Programming Language Design and Implementation, 1991, pp [8] P. Hooimeijer and W. Weimer. Modeling bug report quality, In Proceedings of the 22nd International Conference on Automated software engineering, 2007, pp

8 [9] S. Just, R. Premraj, and T. Zimmermann. Towards the next generation of bug tracking systems, In Proceedings of the 2008 IEEE Symposium on Visual Languages and Human-Centric Computing, September [10] A. J. Ko and B. A. Myers. Debugging reinvented: asking and answering why and why not questions about program behavior. Proceedings of the ICSE-2008, International Conference on Software Engineering, 2008, pp