Automatically PR-Miner: Extracting Implicit Programming Rules and Detecting Violations in Large Software Code
Zhenmin Li and Yuanyuan Zhou
Department of Computer Science University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA
{zli4,
yyzhou}@cs.uiuc.edu
Keywords: automated specication generation, automated violation detection, data mining for software engineering, programming rules, pattern recognition, static analysis
ABSTRACT
Programs usually follow many implicit programming rules, most of which are too tedious to be documented by programmers. When these rules are violated by programmers who are unaware of or forget about them, defects can be easily introduced. Therefore, it is highly desirable to have tools to automatically extract such rules and also to automatically detect violations. Previous work in this direction focuses on simple function-pair based programming rules and additionally requires programmers to provide rule templates. This paper proposes a general method called PR-Miner that uses a data mining technique called frequent itemset mining to efciently extract implicit programming rules from large software code written in an industrial programming language such as C, requiring little effort from programmers and no prior knowledge of the software. Beneting from frequent itemset mining, PR-Miner can extract programming rules in general forms (without being constrained by any xed rule templates) that can contain multiple program elements of various types such as functions, variables and data types. In addition, we also propose an efcient algorithm to automatically detect violations to the extracted programming rules, which are strong indications of bugs. Our evaluation with large software code, including Linux, PostgreSQL Server and the Apache HTTP Server, with 84K3M lines of code each, shows that PR-Miner can efciently extract thousands of general programming rules and detect violations within 2 minutes. Moreover, PR-Miner has detected many violations to the extracted rules. Among the top 60 violations reported by PR-Miner, 16 have been conrmed as bugs in the latest version of Linux, 6 in PostgreSQL and 1 in Apache. Most of them violate complex programming rules that contain more than 2 elements and are thereby difcult for previous tools to detect. We reported these bugs and they are currently being xed by developers. Categories and Subject Descriptors: D.2.4 [SOFTWARE ENGINEERING]: Software/Program Verication Statistical methods; D.2.5 [SOFTWARE ENGINEERING]: Testing and Debugging Debugging aids; D.2.7 [SOFTWARE ENGINEERING]: Distribution, Maintenance, and Enhancement Documentation General Terms: Algorithms, Management, Documentation
1. INTRODUCTION 1.1 Motivation
Programs usually follow many implicit programming rules. A simple example of a programming rule is the function call pair of lock and unlock: a call to lock should be followed by a call to unlock later. Besides such a well-known programming rule, there are many other implicit rules in large software. For example, as shown in Figure 1, PostgreSQL, a well-known open-source database server, contains an implicit programming rule that a call to SearchSysCache must be followed by ReleaseSysCache. The reason is that the function SearchSysCache returns a cache copy of a specied tuple; so after the caller nishes using the tuple, it must call ReleaseSysCache to release it so that this copy can be replaced by other data in the cache. This rule appears 209 times in PostgreSQL code. If some code violates this rule, it causes a memory leak in PostgreSQLs buffer cache. Many programming rules are much more complex. Some rule may contain more than two program elements, with each element being of a different type including function, variable or data type. For example, the programming rule shown in Figure 2, also extracted from PostgreSQL, contains four function calls and one variable. This rule species the correct procedure to replace a tuple. Specically, it requires that, before replacing a tuple using simple heap update, the relation must be rst opened by calling heap openr, and then call CatalogUpdateIndexes to keep the index consistent. Furthermore, after simple heap update, the relation needs to be closed by calling heap close. This rule appears 68 times in PostgreSQL. Some complex rules may indicate variable correlations, i.e. these variables should be accessed together or modied in a consistent manner. For example, Figure 3 shows that, in the Linux code, the two variables ic.command and ic.driver should be accessed together. This rule appears 98 times in Linux.
postgresql-8.0.1/src/backend/catalog/dependency.c: 1733 getRelationDescription (StringInfo buffer, Oid relid) 1734 { 1735 HeapTuple relTup; ...... 1740 relTup = SearchSysCache (...); ...... 1796 ReleaseSysCache (relTup); 1797 }
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. ESEC-FSE05, September 59, 2005, Lisbon, Portugal. Copyright 2005 ACM 1-59593-014-0/05/0009 ...$5.00.
Figure 1: A function-pair rule in PostgreSQL (extracted by PRMiner). This rule appears 209 times in PostgreSQL code.
postgresql-8.0.1/src/backend/commands/tablecmds.c: 5686 AlterTableCreateToastTable(Oid relOid, bool silent) 5687 { ...... 5692 Relation class_rel; ...... 5853 class_rel = heap_openr (...); ...... 5863 simple_heap_update (class_rel, ...); ...... 5866 CatalogUpdateIndexes(class_rel, ...); ...... 5870 heap_close (class_rel, ...); ...... 5891 }
linux-2.6.11/drivers/scsi/3w-9xxx.c: 1964 int __devinit twa_probe(struct pci_dev *pdev, ...) 1965 { 1966 struct Scsi_Host *host = NULL; ...... 1985 host = scsi_host_alloc(...); ...... 2036 scsi_add_host(host, &pdev->dev); ...... 2069 scsi_scan_host(host); ...... 2088 }
(a) Programming rule in twa probe
linux-2.6.11/drivers/ieee1394/sbp2.c: 688 struct scsi_id_instance_data *sbp2_alloc_device (struct unit_directory *ud) 689 { ...... 692 struct scsi_id_instance_data *scsi_id = NULL; ...... 745 scsi_host = scsi_host_alloc(...); ...... 753 if (!scsi_add_host(scsi_host, &ud->device)) { ...... // scsi_scan_host(scsi_host) is missing! 764 }
Figure 2: A complex programming rule containing four functions and one variable in PostgreSQL (extracted by PR-Miner). This rule appears 68 times in PostgreSQL code.
linux-2.6.11/drivers/isdn/hisax/config.c: 771 void ll_stop(struct IsdnCardState *cs) 772 { 773 isdn_ctrl ic; 775 ic.command= ISDN_STAT_STOP; 776 ic.driver = cs->myid; 777 cs->iif.statcallb(&ic); 779 }
Figure 3: A programming rule of variable correlation in Linux. It appears 98 times in Linux code. Implicit programming rules such as those shown above are intrinsic features of programs and violations to these rules can result in software defects. Figure 4 shows such an implicit programming rule and a violation detected by PR-Miner from the latest version of Linux. The rule shown in Figure 4(a) is for probing and initializing a SCSI device: the system should call scsi host alloc to allocate a data structure for the device driver, then call scsi add host to register the device with the SCSI stack, and nally scan the host by scsi scan host. This is the correct procedure for probing SCSI devices in Linux. This rule appears 27 times in Linux. However, there are two program locations missing scsi scan host in the latest version of Linux as shown in Figure 4(b), which are undetected defects in Linux (we reported these two bugs as well as others to the Linux developers and they are being xed now). Such implicit programming rules are useful information for software development. Unfortunately, they usually exist only in programmers minds as they are too tedious to be documented manually. In addition, rule maintenance is a hard task since some rules can change in new versions. Moreover, when the software scales up, the number of rules also increases signicantly. As a result, most of them are undocumented, especially in open-source projects. Consequently, violations to these rules are easy for programmers to introduce, especially for new programmers who are unaware of these rules. Therefore, it is highly desirable if programming rules can be automatically extracted from existing source code. The extracted rules can thereafter be used as a specication to be referenced by programmers. In addition, it is also useful to automatically detect violations to these rules to make software more robust. Previous work [8] by Engler, Chen and Chou has conducted a preliminary investigation in this direction. They proposed a method to extract programming rules using programmer-specied rule templates such as function a must be paired with function b. The two arguments, a and b, will be t by passing all plausible a-b pairs that are selected based on statistical analysis and weighted by naming conventions such as the substrings lock and unlock.
(b) Violation in sbp2 alloc device
Figure 4: An example of a programming rule involving multiple functions in Linux 2.6.11. The rule {scsi host alloc, scsi add host}{scsi scan host} appears 27 times in different functions, one of which is shown in (a). PR-Miner detects two violations that miss the function scsi scan host, which are potential bugs. One of the violations is shown in (b). While the work above is inspiring and proposes a promising direction, it extracts only pair-wise programming rules. Our experimental results indicate that programming rules with 2 elements only account for a small portion (14%) of all rules extracted by PRMiner. In addition, their work requires programmers to give some particular rule templates such as function a must be paired with function b, which not only restricts the types of rules extracted but also needs some specic knowledge about the target software. Furthermore, programmers also need to give different weights to the functions and variables when tting elements into templates. To signicantly advance the state-of-the-art, it would be benecial if implicit programming rules, including complex ones, could be automatically extracted in a general form without requiring prior knowledge or rule templates from programmers. To do this, a naive method is to check every possible combination of program elements to see if they are frequently used together in the target softwares code. Obviously, for large software such as Linux that contains hundreds of thousands of functions and variables, such a naive method would result in exponential complexity. Therefore, an efcient method needs to be developed to achieve the goal.
1.2 Our Contributions
In this paper, we propose a novel method called PR-Miner (Programming Rule Miner), that uses a data mining technique to automatically extract general programming rules from software code written in an industrial programming language such as C 1 and deSimilar to other automatic specication generation tools, we assume that the software has been reasonably well tested and runs correctly most of the time.
1
tect violations with little effort from programmers. More specically, our paper makes the following two major contributions: (1) We propose a general method to automatically extract implicit programming rules from large software code. Beneting from data mining techniques, PR-Miner can extract thousands of programming rules from software such as Linux with 3500 les and a total of 3 million lines of code within 1 minute. Compared with the previous work [8] that extracts only function-pair based rules, PRMiner extracts substantially more rules. This is because PR-Miner has substantially generalized Engler et als work in the following two aspects: General method: Our technique for extracting programming rules is more general. PR-Miner can automatically extract programming rules from software code without any prior knowledge about the software or requiring any annotation, templates or weight assignments from programmers. Additionally, by replacing the frontend parser, PR-Miner can be easily modied to work with programs written in other programming languages such as Java. General rules: The programming rules extracted by PR-Miner are more general. Since it does not limit the programming rules using any xed templates, PR-Miner can extract rules in general forms and with multiple program elements of different types including functions, variables, data types, etc. As a result, it not only extracts simple pair-wise rules, but also extracts more complex rules like the examples shown above. (2) We also propose an efcient algorithm to detect violations to the extracted programming rules. PR-Miner has detected many violations to the extracted rules in the latest versions of Linux and PostgreSQL within 1 minute. Among the top 60 violations reported by PR-Miner, many of them have been conrmed as bugs, including 16 bugs in Linux, 6 bugs in PostgreSQL and 1 bug in Apache. These bugs are currently being xed by corresponding developers after we reported. Most of these bugs are semantic bugs that violate complex programming rules that contain more than 2 elements and are thereby difcult for previous tools to detect. The rest of this paper is organized as follows. Section 2 briey describes the background of the data mining technique used in PRMiner. Section 3 presents how PR-Miner automatically extracts implicit programming rules and detects violations. Section 4 presents the evaluation results, followed by discussion of PR-Miners limitations in Section 5. Section 6 presents related work and Section 7 concludes the paper.
To solve the frequent itemset mining problem, quite a few algorithms have been proposed. PR-Miner uses a FP-tree-based mining algorithm called FPclose [14], which is one of the most efcient frequent itemset mining algorithms. Instead of generating the complete set of frequent sub-itemsets, FPclose mines only the closed sub-itemsets. A closed sub-itemset is the sub-itemset whose support is different from that of its super-itemsets. In the example above, the frequent sub-itemsets {b}, {d}, {a, b}, {a, d} and {b, d} are not closed since their supports are the same as their superitemset {a, b, d}. FPclose only generates the closed sub-itemsets {a}:4 and {a, b, d}:3 as result. This can signicantly improve time and space performance since it can avoid generating exponential number of frequent sub-itemsets. After all closed frequent sub-itemsets are mined from an itemset database, association rules can be generated. An association rule can be denoted as X Y with condence c and support s, where X and Y are itemsets. The meaning of the rule is that if an itemset contains X, it also contains Y with probability of c [1, 15]. Association rules allow violation detection. If the condence is very high, say 99%, the itemset that contains only X but not Y violates the rule, indicating a potential outlier. Due to space limitation, we do not describe the details of the FPclose algorithm since they can be found in [14].
3. PR-Miner
PR-Miner has two major functionalities: automatically extracting implicit programming rules, and automatically detecting violations to the extracted programming rules. The owchart of PRMiner is shown in Figure 5. This section rst gives an overview of PR-Miner, and then presents how PR-Miner automatically extracts programming rules from source code, and how it detects rule violations and prunes false positives.
3.1 Overview
The high-level idea of PR-Miner in automatic rule extraction is to nd associations among elements (e.g., function, variable, data type) by looking for elements that are frequently used together in source code. For example, calls to spin lock irqsave and spin unlock irqrestore in Linux appear together within the same function for more than 3600 times, which indicates that spin unlock irqrestore following spin lock irqsave is very likely to be an implicit programming rule. By identifying which elements are used together frequently in the source code, such correlated elements can be considered a programming rule with relatively high condence. Of course, as described in the introduction, a naive implementation of this high-level idea is infeasible since it needs to examine all possible element combinations. In order to efciently nd program element correlations, PRMiner converts the problem into a frequent itemset mining problem by rst parsing the software source code as shown in Figure 5. Each program element is hashed into a number, then a function denition is mapped into an itemset (a set of numbers), which is written as a row into the itemset database. As a result, the whole program is converted into a database that contains many itemsets. By mining this database using a frequent itemset mining algorithm such as FPclose, we can nd the frequent sub-itemsets that appear for many times. These frequent sub-itemsets can then be used to infer programming rules. For a frequent sub-itemset discovered by the mining algorithm, we call the set of the corresponding program elements a programming pattern, which indicates that the program elements are correlated and frequently used together. For example, FPclose can nd that {spin lock irqsave, spin unlock irqrestore} is
2.
BACKGROUND OF DATA MINING
PR-Miner is based on a data mining technique called frequent itemset mining [1, 14], which has broad applications, including mining motifs in DNA sequences, analysis of customer shopping behavior, etc. The goal of frequent itemset mining is to efciently nd frequent itemsets in a large database, where an itemset is a set of items. In a database composed of a large number of itemsets, if a sub-itemset (subset of an itemset) is contained in more than a specied threshold (called min support) of itemsets, it is considered frequent. The number of occurrences of a sub-itemset A is denoted as its support. The itemset that contains A is called its supporting itemset. For example, in an itemset database D: D = {{a, b, c, d, e}, {a, b, d, e, f }, {a, b, d, g}, {a, c, h, i}} The support of sub-itemset {a, b, d} is 3, and its supporting itemsets are {a, b, c, d, e}, {a, b, d, e, f } and {a, b, d, g}, If min support is specied as 3, the frequent sub-itemsets for D are {a}:4, {b}:3, {d}:3, {a, b}:3, {a, d}:3, {b, d}:3 and {a, b, d}:3, where the numbers are the supports of the corresponding sub-itemsets.
a programming pattern since it appears in the Linux source code more than 3600 times. Note that programming patterns are different from programming rules. For example, the above pattern may lead to one or two of the following programming rules: {spin lock irqsave}{spin unlock irqrestore} {spin unlock irqrestore}{spin lock irqsave} The rst rule says that whenever the program calls spin lock irqsave, it should also call spin unlock irqrestore, while the second says that whenever the program calls spin unlock irqrestore, it should also call spin lock irqsave. These two are different rules, and not all of them denitely hold, even if the pattern has appeared for many times. Therefore, after programming patterns are extracted using the frequent itemset mining technique, PR-Miner needs to generate programming rules from the extracted patterns. The main idea of the rule generation process is to nd the number of cases that contain the items on the left but not those on the right. For example, in the above example, we need to nd out how many cases that spin lock irqsave appears but spin unlock irqrestore does not and vice versa. After generating the programming rules, PR-Miner stores them in specication les so that programmers can examine them and also use them later as references. Section 3.3 describes the rule generation process in detail. After programming rules are generated, PR-Miner automatically detects violations in source code. It also automatically prunes false positives and ranks violations in the report so that programmers only need to examine top ranked violations. The violation detection process is based on the idea that a programming rule is usually followed in most cases and violations occur only in a small percentage of cases. Section 3.4 describes the detection process in more detail. Using closed frequent itemset mining algorithms such as FPclose provides PR-Miner several benets: (1) Generality. Close frequent itemset mining algorithms do not limit the number of items in frequent sub-itemsets and also does not require any rule templates. Furthermore, the items in a frequent sub-itemset are not necessarily adjacent in the supporting itemset, i.e. they can be far apart from each other. (2) Time efciency. Data mining algorithms such as FPclose are usually very efcient since they strive to avoid scanning data too many times by eliminating redundant computation as much as possible. Additionally, since FPclose generates only closed frequent itemsets, it can avoid generating an exponential number of sub-itemsets. (3) Space efciency. From closed frequent sub-itemsets, we can nd closed rules, rules that subsume many other rules with the same support. Take the itemset database D = {{a, b, c, d, e}, {a, b, d, e, f }, {a, b, d, g}, {a, c, h, i}} described in Section 2 as an example. FPclose nds a closed frequent subitemset: {a, b, d}:3. From the closed frequent sub-itemset we can have the following 6 closed rules with support 3: {a} {b, d} with condence 3/4=75% {b} {a, d} with condence 100% {d} {a, b} with condence 100% {a, b} {d} with condence 100% {a, d} {b} with condence 100% {b, d} {a} with condence 100% Other rules are subsumed by the above closed rules. For example, the sub-rule {a} {b} with condence 75% is subsumed by the rst closed rule. Using closed rules not only saves space, but also signicantly reduces the number of rules that need to be examined or referenced by programmers. In addition, it also speeds up the violation detection process (See Section 3.4).
Source files Parsing & hashing Itemsets Mining Programming patterns Generating programming rules Storing rules into specification files Examined & referenced by programmers Detecting violations Pruning false violations Ranking Potential bugs
Figure 5: Flowchart of PR-Miner
3.2 Extracting Programming Patterns
3.2.1 Parsing Source Code
The main purpose of parsing source code is to build an itemset database in order to convert the programming pattern extraction problem into a frequent itemset mining problem. PR-Miner does this by using a modied GCC compiler [25] as the parser to convert each function denition into a set of numbers. The current prototype of PR-Miner only works for C, but it can be easily extended to other programming languages by replacing the GCC front ends. In order to convert the source code into an itemset database, we need to address the following issues: (1) How to parse the source code? (2) What elements in the source code should be converted? (3) How to represent elements using numbers? To parse the source code, PR-Miner rst uses the GCC front end to obtain the intermediate representation. The intermediate representation is stored in a tree data structure, with each node representing various types of elements in source code including identier name, data type name, keyword, operator, control structure, and so on. In order to convert a function to an itemset, PR-Miner traverses the representation tree of this function, and hashes each selected elements to a number. By combining the hash values of all selected elements in a function, this function is mapped to an itemset. Then the itemsets of all functions construct the itemset database to input to the mining algorithm FPclose. The reason to convert a function instead of a basic block to an itemset is that most programming rules usually occur within the scope of a function. Of course, some rules can span across multiple functions. But mining these rules is much harder as it requires deeper inter-procedural analysis, so extracting such rules remains as our immediate future work. Not every program element in the intermediate representation is converted into a number because some elements can cause noise. For example, keywords and simple data types such as int appear in almost every function. They are less likely lead to interesting programming rules. In addition, including them in the itemset would signicantly increase the computation of frequent itemset mining. Therefore, PR-Miner does not hash such elements into numbers. Furthermore, the same programming rule involving local variables may use different variable names at different code segments. For instance, in the example shown in Figure 4, the return value of calls to the same function scsi host alloc can be assigned to different local variables such as host and scsi host. If we hash them into different numbers, the rule might be missed. In order to catch such kinds of rules, we need to use the common characteristics of these local variables such as their data types to represent them so that they are still hashed to the same number in the itemset
database. For example, the local variable class ref in the code segment in Figure 2 is represented by the hash value of its data type Relation instead of its name class ref. Another problem when hashing identiers to numbers is name collisions. Different types of identiers with the same name would be hashed to the same number, causing false positives in the generated frequent sub-itemsets. In order to eliminate such name collisions, PR-Miner hashes different types of identiers into different values. To do that, PR-Miner rst prexes every identier name based on its type, and then hashes the prexed name to a number. For example, a function call to lock would be prexed with F- and then be hashed to a number corresponding to F-lock, while a global variable with the same name lock would be prexed with G- and be hashed to a number corresponding to G-lock. Similarly, different record structures may use the same name for their elds, which is quite common in large software. For example, the names, next and prev are commonly used as eld names in many different structures. Such name collisions would result in false positives of frequent sub-itemsets. In order to differentiate elds of the same name but in different record structures, PR-Miner attaches the associated record type to every eld name. For example, the elds next in the record types tree and list are considered as D-tree.R-next and D-list.R-next, respectively, and so they can be hashed into different numbers to avoid collision. The hash function PR-Miner uses is hashpjw [2], chosen for its low collision rate. Our experiments show that its collision rate is low enough for frequent sub-itemset mining. Additionally, if conict-free mapping is needed, we can rst parse the whole source code so that we can create a symbol table for all possible identiers, and then convert elements into numbers based on their indexes to the symbol table. Since it takes one more pass of the source code and our hashing method already has a low collision rate, we do not use this method. Table 1 shows how PR-Miner converts a function into an itemset. After parsing the source code, prexing and hashing selected elements into numbers, PR-Miner converts the denition of function twa probe into the itemset {92, 39, 41, 68, 56, 36, ...}.
Source code Preprocessed identifiers Hash values 92 ...... 92 39 41 ...... 68 92 56 ...... 36 92 ......
example shown in Table 1. For simplicity, let us denote these three functions as add, alloc and scan. The sub-itemset {39, 68, 36, 92} appears in totally 27 itemsets in the itemset database converted from Linux code. Suppose that min support is set as 15. FPclose will nd a frequent sub-itemset {39, 68, 36, 92} with a support of 27, which means that the corresponding functions alloc, add and scan, and the data type Scsi Host are used together for 27 times. Therefore, these four elements are correlated with each other and are thereby outputted as a programming pattern, which is then used to generate programming rules in the next step. Since FPclose generates only closed frequent itemsets whose support is larger than the support of its super-itemset, it does not generate redundant sub-patterns with the same support. In the above example, {39, 68, 36} is also a frequent sub-itemset. However, since it is not closed, i.e., it is included in its super-itemset {39, 68, 36, 92} with the same support 27, we do not need to output it. It is not enough to know only the closed programming patterns and their support values (i.e., how many times the pattern occurs). It would be more helpful for programmers if we also record the functions in which each extracted pattern occurs. Such information is also needed later in violation detection in order to know which function violates an extracted rule. Unfortunately, the original algorithm FPclose and any other frequent itemset mining algorithms were not designed exactly for our purpose. They only output the support values for each discovered pattern but not their supporting itemsets. Therefore, we enhance FPclose to address this problem by also maintaining the supporting itemsets during the mining process. In the above example, PR-Miner outputs the closed frequent sub-itemset {39, 68, 36, 92} with the supporting itemset that corresponds to the 27 functions that contain this programming pattern.
3.3 Generating Programming Rules
As we explained briey in Section 3.1, extracting only programming patterns is not enough because a pattern may lead to many different rules. Therefore, we also need to generate rules from patterns based on conditional probabilities.
3.3.1 A Naive Method
A naive method to generate programming rules from extracted patterns is to divide the items in each closed frequent sub-itemset into two parts and then calculate the condence. In other words, from a closed frequent sub-itemset I, we can compute the condence for every possible association rules X Y , where X and Y are subsets of I [1, 15]. The support of such a rule is equal to the support of I, while the condence of a rule is the conditional probability, i.e. support(I)/support(X), where support(X) is the number of occurrences of sub-itemset X in the itemset database, which also equals to the maximum support of any closed frequent itemset that contains X. Basically, the condence indicates the conditional probability that if X occurs, the likelihood for Y to occur. Rules with condence smaller than a specied threshold (e.g. 90%) are pruned. And the remaining rules are outputted to the specication les to be examined and referenced by programmers. Let us consider the above example again. After PR-Miner nds a programming pattern {alloc, add, scan, Scsi Host}. From this pattern, the naive method can generate 14 different possible rules by partitioning these three functions and the data type into 2 subsets in all possible ways such as {add}{alloc, scan, Scsi Host}, and {add, alloc}{scan, Scsi Host}, and so forth. All these rules have the support of 27. From the programming patterns discovered by FPclose, we know that the support for {add} is 37, and the support for {add, alloc} is 29. Therefore, the condences for these 2 rules are 27/37 = 72.9%
linux-2.6.11/drivers/scsi/3w-9xxx.c: L1964 - 2088 T-Scsi_Host ...... T-Scsi_Host int __devinit twa_probe(struct pci_dev *pdev,...) F-scsi_host_alloc { T-scsi_host_template struct Scsi_Host *host = NULL; ...... ...... F-scsi_add_host host = scsi_host_alloc(&driver_template, ...); T-Scsi_Host ...... T-pci_dev.R-dev retval = scsi_add_host(host, &pdev->dev); ...... ...... F-scsi_scan_host scsi_scan_host(host ); T-Scsi_Host ...... ...... }
Table 1: Example of parsing a function. The italic identiers in source code are selected to analyze. They are prexed with the types as shown in the second column. Each preprocessed identiers is then hashed to a number. Only the last two digits of hash values are shown for simplicity.
3.2.2 Mining for Programming Patterns
After PR-Miner parses the source code and generates an itemset database, it applies the closed frequent itemset mining algorithm, FPclose, on the database to nd closed frequent sub-itemsets. As we describe in Section 2, if a set of numbers appear together in any itemsets for more than a specied threshold number (min support) of times, this sub-itemset is considered frequent. Let us consider the
and 27/29 = 93.1%, respectively. The condences for the other 12 rules can also be computed similarly. So if we set the condence threshold to be 90%, the rst rule {add}{alloc, scan, Scsi Host} is pruned, while the second rule is outputted. The biggest problem with the naive method is that it needs to examine all possible rules from each mined patterns. A programming pattern with k elements can generate up to (2k 2) rules, which is impractical for long patterns. For example, our evaluation with large software code shows that some programming patterns are composed of more than 20 elements. Therefore, it is time and space inefcient to use this naive method to generate rules from patterns.
3.3.2 Generating Closed Rules
Instead of examining all possible programming rules from a mined pattern like in the naive method, PR-Miner examines only closed rules. As we explained in Section 3.1, it is enough to generate only closed rules since other rules are subsumed by closed rules. To further reduce the number of outputted rules and speed up the generation as well as the violation detection processes, PR-Miner stores closed rules in condensed format. Formally, the condensed format for a closed frequent sub-itemset I is: I : s|{C1 : s1 |s1 > s} . . . {Cm : sm |sm > s} where C1 . . . Cm are all subsets of I whose supports (s1 . . . sm ) are different from Is. Obviously, s1 . . . sm are all larger than s. Such condensed format can represent all the closed rules derived I from and their condences can be computed easily. For a closed rule X Y derived from I, if X equals to Ci (i.e. a subset of I with a support larger than I), the condence of the rule is s/si ; otherwise, the condence of the rule is 100%. For example, suppose FPclose extracts two closed frequent subitemsets: {a} : 4 and {a, b, d} : 3. The condensed format that represents all the closed rules derived from {a, b, d} is {a, b, d} : 3|{a : 4} It explicitly expresses that the rule {a} {b, d} has condence 3/4=75%, and also infers that any of the other 5 closed rules, such as {a, b} {d} has condence 100%. Now the rule generation problem becomes how to nd out all of the subset Ci that has a support si larger than s. Since the support of Ci is larger than s, it indicates that Ci should be contained in another closed frequent sub-itemset (based on the denition of closed frequent sub-itemset). Since Ci may include in multiple other closed frequent sub-itemset, PR-Miner needs to nd the one with the maximum support. To achieve this goal, PR-Miner uses a clever idea that converts this problem back to a frequent sub-itemset mining again. In other words, PR-Miner uses FPclose one more time to nd common subitemsets from frequent sub-itemsets generated by the rst pass of FPclose. Doing such will nd all common subsets among the closed frequent sub-itemsets generated in the rst pass. Let CommonSub denote all the common subsets generated by the second pass of FPclose. If a subset Ci of I is included in CommonSub, we can immediately nd out which super-itemset of Ci has the maximum support. The support of this super-itemset must be equal to the support of Ci based on the denition of closed frequent sub-itemsets. We can easily prove this by contradiction (The proof is omitted due to space limitation). Note that the basic operation we need is to compute the common subsets for each pair of the closed frequent sub-itemsets. Therefore, we can apply the frequent itemset mining algorithm again on the closed frequent sub-itemsets with minimum support of 2. Our algorithm C LOSED RULES for generating closed rules in condensed format is shown in Figure 6.
Algorithm: C LOSED RULES(I) Input: I = {Ik |1 k n}, Ik has 3 elds Fk , sk , Ek ; Output: The closed rules R in condensed format. 1: Sort I by supports in descending order such that s1 s2 ... sn 2: Mine common closed frequent sub-itemsets from I: C FP CLOSE({Fi |i = 1, 2, ..., n}, 2), where C = {Ci |1 i m} and Ci has 3 elds Fi , si , Ei 3: for i = 1, 2, ..., m 4: Denote Ei = {ij |1 j si } 5: for j = 2, 3, ..., si 6: if si1 > sij 7: Insert Fi : si1 to sub-itemset Iij in R Figure 6: Generating closed rules R in condensed format from
closed frequent itemsets I mined from the rst step explained in Section 3.2. The close frequent mining algorithm FP CLOSE takes an itemset database and the minimum support threshold as input, and outputs the closed frequent sub-itemsets, each of which has three elds Fi , si , Ei , where Fi is the frequent itemset itself, si is its support, Ei is the indexes of its supporting itemsets, and Ei is sorted in an ascending order. Similarly, Fi , si , Ei have the same meanings but are generated by the second pass of FP CLOSE (line 2) to a database that consists of all closed frequent sub-itemsets, i.e. {Fi |i = 1, 2, ..., n}.
In the C LOSED RULES algorithm, it rst sorts the frequent itemsets I mined from FPclose (line 1) so that it can quickly locate the frequent itemset with the maximal support for any common subitemset. In line 2, it calls FPclose with minimum support of 2 to nd out all common sub-itemsets C from I. For each common subitemset Ci (line 3), C LOSED RULES inserts it with its support to the corresponding rule of condensed format as follows. Ei includes the indexes of all Ci s supporting itemsets in I. The rst supporting itemset Ii1 has the maximum support for Ci , because all indexes in Ei are sorted based on their corresponding itemsets support. For the other supporting itemset Iij (line 5), if its support sij is smaller than si1 (line 6), Ci is inserted into the subset of the rule for the closed frequent itemset Iij . This way, with only one pass it can insert Ci into all rules that are super-itemsets of Ci but have smaller support than Ci . C LOSED RULES performs much better than the naive algorithm in terms of space and time, because it does not need to examine all possible rules generated from extracted programming patterns. By calling C LOSED RULES on the closed frequent sub-itemsets that correspond to the extracted programming patterns, PR-Miner obtains the closed rules in the condensed format expressed in numbers, and then it maps the closed rules back to programming rules and stores them into a specication le. The programmers can then validate the programming rules so that later they can use them as specications, and also new programmers can read them when they start coding to avoid mistakes. Since PR-Miner extracts programming rules based on occurrences, some false positives may be introduced if some elements only coincidentally appear together for many times in the source code. However, a rule with larger supports can be more believable. Therefore, PR-Miner ranks the rules based on supports: programmers can examine those rules that are ranked in the top 100 or 500. Furthermore, as we explained early, rules with condence lower than the specied threshold (e.g. 90%) are pruned. Additionally, some other ranking method such as giving weights for different elements as in Engler et als work [8] can also be applied.
3.4 Detecting Violations to Extracted Rules
Based on the programming rules generated from the previous step, PR-Miner can nd potential bugs by detecting violations to these rules. The main idea is that the programming rules usually hold for most cases and violations happen only occasionally. Take the potential bug detected by PR-Miner in Figure 4 as an example. The function call to scan should follow alloc and add as the programming rule indicates. This rule appears 27 times in Linux, but there are 2 cases violating this rule because scan is missing. As shown in Figure 5, PR-Miner rst detects violations to the extracted programming rules, then prunes the false violations using inter-procedural analysis, and nally ranks the violations in the error report.
F F1 F11 ... ... Fn
F-11 ...
F-1 ... F-1n ... ...
...
F F-p1 ...
F-p ... F-pm ... ...
... ... ... Fnm
(a) Checking in callees
(b) Checking in all callers
Figure 7: Checking the call paths for pruning false violations In order to prune the above false violations, PR-Miner performs an inter-procedural checking. It rst checks the callees paths for each function that contains violations. For each violation of rule X Y in function F , PR-Miner checks whether every item y Y is in the functions F1 , ..., Fn called by F . As shown in Figure 7(a), we can follow the calling path more deeply by checking the functions called by callees F1 , ..., Fn in F . If the missing items are in any of the calling paths, it is a false violation. For time efciency, PR-Miner limits the checking depth. Since PR-Miner outputs all function calls in each function denition as described in Section 3.2.1, it is easy to follow the calling path during checking. Besides callees, PR-Miner also checks the callers to prune false positives. In the example above, there is also a violation in the function try lock because lock is in try lock but unlock is not in it. In order to prune such false violations, PR-Miner also checks whether the missing items are in the caller functions paths as shown in Figure 7(b). In order to check the call path backwards, PR-Miner maintains a caller list for each function F that consists of the indexes of the functions that call F. If the missing items for function F are in the paths of all of its callers, it is a false violation.
3.4.1 Detecting Violations
In order to detect violations to the programming rules, a naive method is to generate all possible programming rules and then check them upon the source code one by one. As we discussed in Section 3.3, there would be an exponential number of rules that need to be checked. Fortunately, it is unnecessary to check all programming rules. First, if the rule has a low condence, it is already pruned in the rule-generation step. In other words, if the condence threshold is t, any rules with condence smaller than t are discarded. Second, if the rule has 100% condence, it indicates that there is no violation for the rule. Therefore, we only need to check the rules with condence in the range [t, 100%). The main idea of the violation detection process is straightforward. For example, if a rule {a, b} {d} has a support of 100 and {a, b} has a support of 101, there is only one out of 101 cases that has {a, b} but not {d}, which indicates that this case violates the rule {a, b} {d}. In other words, this case is likely to be a bug. But if {a, b} has a support of 200, the rule {a, b} {d} will be pruned as its condence is only 50%. Since PR-Miner stores generated programming rules in the condensed format that explicitly indicates which rules have condence less than 100% but greater than the specied threshold t, we can easily gure out which rules have violations. Even more efciently, PR-Miner detects violations during the same process when it generates the programming rules by calling C LOSED RULES as shown in Figure 6. To do that, PR-Miner computes the condence for the rule Fi (Fij Fi ) in the loop of line 5 as c = sij /si1 . If t c < 1, it indicates that there are violations to this rule. The violations can be easily gured out by comparing the supporting itemsets for the closed frequent sub-itemsets Ii1 and Iij as follows. Fi1 contains the common sub-itemset Fi , but it does not contain (Fij Fi ). It means that some supporting itemsets in Ei1 violate the rule Fi (Fij Fi ). On the other hand, this rule is supported by the supporting itemsets Eij for Fij . Therefore, the itemsets in Ei1 but not in Eij violate this rule, and so the corresponding functions of the itemsets violate the programming rule.
3.4.3 Ranking and Reporting Bugs
After PR-Miner detects rule violations and prunes false positives, it ranks all remaining violations and reports them to programmers. PR-Miner ranks the violations based on the condence of the violated rules. Since a function may contain several violations, PRMiner groups all violations of the same function together, and the violation with the highest condence is assigned as the condence of the violated function. The condence of a violated function can be considered the possibility that the function has bugs. In the current version of PR-Miner, it simply ranks the bugs by the condence. Because several functions may have the same violation, the potential bugs in these functions are strongly correlated. Therefore, some other advanced ranking schemes such as correlation ranking [17] can be used here to further improve the accuracy of our ranking function, which remains as our future work.
4. EVALUATION 4.1 Experiment Setup
We have evaluated PR-Miner with the latest versions of Linux, PostgreSQL, and the Apache HTTP Server. The numbers of les, lines of code (LOC) and functions are shown in Table 2. PR-Miner takes three parameters: min support, the condence threshold, and maximal checking path depth. By default, we set the min support as 15, the condence threshold as 90%, and the maximal depth of call path as 3 for pruning false violations. The parser for PR-Miner is GCC 3.3.4 [25] with a small modications. In our experiments, we run PR-Miner on an Intel Xeon 1.5 GHz machine with 4GB memory and Linux 2.4.20 system.
Software Linux PostgreSQL Apache version 2.6.11 8.0.1 2.0.53 #C les 3,538 409 160 LOC 3,037,403 381,192 84,724 #functions 73,607 6,964 1,912
3.4.2 Pruning False Violations
The violation detection above can result in false positives if the elements in a programming rule span across multiple functions. The reason is that PR-Miner detects violations using only intraprocedural analysis because, as described in Section 3.2, each itemset in the database corresponds to a function denition. Suppose in an example with a function-pair (lock and unlock) rule, unlock is called inside a function F but lock is not. Instead, F calls another function try lock that calls lock. Without interprocedural analysis, PR-Miner would report that F contains a violation of missing lock, even though F contains lock in its callee.
Table 2: Software evaluated in our experiments.
4.2 Extracting Implicit Programming Rules
Table 3 shows the number of closed rules discovered by PRMiner in the evaluated software. Rules that have condence lower than the threshold (90%) are pruned automatically by PR-Miner and are thereby not included in the results reported in this section. From the closed rules, programmers can easily infer other rules subsumed by these closed rules. These closed rules can be classied into three categories: function-function (F-F) rules, variablevariable (V-V) rules, and function-variable (F-V) rules. F-F rules involve only functions, V-V rules involve only variables (including elds in structure) or their data types, while F-V rules involve both functions and variables.
Software Linux PostgreSQL Apache Total 32,283 6,128 283 F-F 1,075 379 33 V-V 8,883 687 92 F-V 22,325 5,062 158
Figure 8: Distribution of rule support in the Linux code. (Y-axis is in logarithmic scale)
5000
Linux
#Rules
4000 3000 2000 1000 0
Our results show that a large number of implicit, undocumented programming rules can be effectively extracted from source code by PR-Miner without any priori knowledge or annotations/specications from programmers. For example, PR-Miner extracts a total of 32,283 implicit, undocumented closed programming rules from Linux. It would be very difcult for programmers to manually specify these many programming rules. PR-Miner effectively relieves such burden from programmers by efciently and automatically extracts such rules from source code. The results also show that around 88.396.7% rules involve variables. For example, there are around 9000 V-V rules in Linux, along with a large number of variable correlations contained in F-V rules. Comparing with the previous studies such as Engler et als work [8] that do not consider rules about variable correlations, PR-Miner can extract substantially more programming rules.
Figure 9: Distribution of rule size in the Linux code. The above results, along with the results shown on Table 3, indicate the generality of PR-Miner over the previous work [8] because PR-Miner does not constrain the rule format or limit the number of elements in the rules to only 2.
4.3 Detecting Violations
PR-Miner has reported many violations of programming rules in the evaluated software. We have manually examined the top 60 violations to differentiate bugs from false positives. Conrmed bugs have been reported to the corresponding developer community and are currently being xed by developers. The numbers of the veried bugs are shown in Table 4. Currently, we are still inspecting the violation report and more bugs will be conrmed. More specifically, we have validated 16 bugs in Linux, 6 bugs in PostgreSQL, and 1 in the Apache HTTP Server. Almost all of these bugs are semantic bugs instead of those simple bugs such as buffer overow, data races, etc. and are thereby difcult to be detected by existing bug detection tools. In addition, most of these bugs violate complex rules that involve more than 2 elements, so it is difcult for the previous work [8] to detect them. Notice that we directly apply the programming rules mined by PR-Miner in violation detection without having programmers validate the extracted rules. Therefore, false programming rules might result in false positives in violation report. If programmers can validate those topped ranked rules and prune false rules, the number of false positives generated by PR-Miner in violation detection should be smaller than those presented in Table 4. Even though our inter-procedural pruning method can prune a lot of false positives in violation report, many false positives still exist. Even for the strong function-pair rules with high condence such as lock-unlock, there are still a few violations that are false positives. For example, the function spin lock bh (in Linux kernel/spinlock.c) includes a call to spin lock irqsave but not spin unlock irqrestore because spin lock bh is to provide locking functionality and thereby does not need unlocking in it. Such false positives can be pruned if we can also conduct deeper inter-procedural analysis. In addition, combining with some dynamic checking methods would be helpful to further prune these false positives, which remains as our future work.
4.2.1 Supports of Programming Rules
Figure 8 shows the support distribution of closed rules extracted by PR-Miner in Linux. As expected, the number of closed rules decreases when the corresponding support increases. The decreasing rate is approximately exponential from 15 to 80 (notice that Y-axis is in logarithmic scale). Since the rules with larger support are more believable, programmers can increase min support to improve the quality of rules, or choose only those top ranked closed programming rules (Extracted rules are ranked by their supports). The gure also shows that some rules have large supports, strongly validating our observation that programmers follow many implicit programming rules in writing software. For example, there are 1442 rules with supports larger than 100 in Linux, and the rule with the largest support is the function pair of spin lock irqsave and spin unlock irqrestore, which has a support of 3656.
4.2.2 Rule Size
Each programming rule contains several elements such as functions, variables and data types. The number of elements in a rule is called the rule size. Figure 9 shows the distribution of rule size in Linux. Around 4200 closed rules contain only 2 elements, which accounts for 14% of all closed rules. On the other hand, 9% of the closed rules have even more than 10 elements. For example, PRMiner found a rule that contains 12 program elements and appears 38 times in Linux, which is followed when the system registers for a PCMCIA device.
2 3 4 5 6 7 8 9 10 1115 1620 >20 Size of Rules
Table 3: The number of closed rules extracted by PR-Miner. Notice that all F-V rules that contain more than 2 elements also include F-F and/or V-V rules as their sub-rules.
However, even these false positives are still useful for automatic specication and annotation of function interfaces. In the example above, we can know the unlock function should be called somewhere after calling spin lock bh. Therefore, we can automatically annotate the function spin lock bh with such assumption. There are many cases that are more complex than this example. For example, PR-Miner reports a violation to a rule that says: if elds counter and len in a structure sk buff are modied, the function kfree skb should be called. This rule appears 480 times in Linux. It indicates that if these two elds are accessed, some memory is allocated for the data structure of sk buff and thereafter it should be freed. However, there is one violation of this rule in the function skb clone in the le net/core/skbuff.c with condence 480/481 = 99.8%. Although it is a false violation, it still indicates that after the function skb clone is called, kfree skb also should be called later in order to free the memory; otherwise, it would cause memory leak. Therefore, this violation can be used to automatically annotate the interface of the function skb clone.
Software Bugs Linux 16 PostgreSQL 6 Apache 1 Inspected (top 60) Specication False Positives 20 24 9 45 0 6 Uninspected 1387 87 0
can combine with our previous work on copy-paste detection called CP-Miner [18], which also works with large software. Using CPMiner, we can rst identify copy-pasted code, and then each group of copy-pasted code accounts as only 1 support in PR-Miner when extracting programming rules and detecting violations. Noisy Effects of Macros. Macro denitions in C can result in false programming rules as well as false negatives in detecting violations. Since GCC rst preprocesses the source code by expanding macros before creating the intermediate representation, the information in a single macro can be duplicated for man...