Download course a comparison of bug finding tools for Java

1 Introduction

In recent years, many tools have been developed for automatically finding bugs in program source code, using techniques such as syntactic pattern matching, data flow analysis, type systems, model checking, and theorem proving. Many of these tools check for the same kinds of programming mistakes, yet to date there has been little direct comparison between them. In this paper, we perform one of the first broad comparisons of several Java bug-finding tools over a wide variety of tasks.
In the course of our experiments, we discovered, some what surprisingly, that there is clearly no single “best” bug finding tool. Indeed, we found a wide range in the kinds of bugs found by different tools (Section 2). Even in the cases when different tools purport to find the same kind of bug,we found that in fact they often report different instances.

Threats to Validity
There are a number of potential threats to the validity of this study. Foremost is simply the limited scope of the study,both in terms of the test suite size and in terms of the selec-tion of tools. We believe, however, that we have chosen a representative set of Java benchmarks and Java bug finding tools. Additionally, there may be other considerations for tools for languages such as C and C++, which we have not studied. However, since many tools for those languages use the same basic techniques as the tools we studied, we think that the lessons we learned will be applicable to tools for those languages as well.
Another potential threat to validity is that we did not ex-actly categorize every false positive and false negative from the tools. Doing so would be extremely difficult, given the large number of warnings from the tools and the fact that we ourselves did not write the benchmark programs in our study. Instead, in Section 4.1, we cross-check the results of the tools with each other in order to get a general sense of how accurate the warnings are, and in order to under-stand how the implementation techniques affect the gener-ated warnings. We leave it as interesting future work to check for false negatives elsewhere, e.g., in CVS revision histories or change logs.

2 Background

A Small Example
The code sample in Figure 1 illustrates the variety and typical overlap of bugs found by the tools. It also illustrates the problems associated with false positives and false nega-tives. The code in Figure 1 compiles with no errors and no warnings, and though it won’t win any awards for function-ality, it could easily be passed off as fine. However, four of the five tools were each able to find at least one bug in this program. (Bandera wasn’t tested against the code for reasons explained later.)
PMD discovers that the variable y on line 8 is never used and generates an “Avoid unused local variables” warn-ing. FindBugs displays a “Method ignores results of In-putStream.read()” warning for line 12; this is an error be-cause the result of InputStream.read() is the num-ber of bytes read, and this may be fewer bytes than the pro-grammer is expecting. FindBugs also displays a “Method
1 import java.io.*;
2 public class Foo{
3 private byte[] b;
4 private int length;
5 Foo(){ length = 40;
6 b = new byte[length]; }
7 public void bar(){
8 int y;
9 try {
10 FileInputStream x =
11 new FileInputStream(« z »);
12 x.read(b,0,length);
13 x.close();}
14 catch(Exception e){
15 System.out.println(« Oopsie »);}
16 for(int i = 1; i <= length; i++){
17 if (Integer.toString(50) ==
18 Byte.toString(b[i]))
19 System.out.print(b[i] + » « );
20 }
21 }
22 }

Java Bug Finding Tools
Figure 2 contains a brief summary of the five tools we study in this paper, and below we discuss each of them in more detail.
FindBugs [13] is a bug pattern detector for Java. Find-Bugs uses a series of ad-hoc techniques designed to balance precision, efficiency, and usability. One of the main tech-niques FindBugs uses is to syntactically match source code to known suspicious programming practice, in a manner similar to ASTLog [7]. For example, FindBugs checks that calls to wait(), used in multi-threaded Java programs, are always within a loop—which is the correct usage in most cases. In some cases, FindBugs also uses dataflow analy-sis to check for bugs. For example, FindBugs uses a sim-ple, intraprocedural (within one method) dataflow analysis to check for null pointer dereferences.
FindBugs can be expanded by writing custom bug detec-tors in Java. We set FindBugs to report “medium” priority warnings, which is the recommended setting.

…