Evaluating Path-sensitive Error Detection Techniques: Challenges and Opportunities

Matt Dwyer

Henson Chair of Software Engineering
Department of Computer Science and Engineering
University of Nebraska, Lincoln

November 30, 2006
Talk: 3:30 pm - 4:30 pm
3105 Engineering
Host: L. Dillon

Recent advances in static program analysis have made it possible to detect errors in applications that have been thoroughly tested and are in wide-spread use. The ability to find errors that have eluded traditional validation methods is due to the development and combination of sophisticated algorithmic techniques that are embedded in the implementations of analysis tools. Evaluating new analysis techniques is typically performed by running an analysis tool on a collection of programs - enabling and disabling a given technique in different runs. While seemingly sensible, this approach runs the risk of attributing improvements in the cost-effectiveness of the analysis to the technique under consideration, when those improvements may actually be due to details of analysis tool implementations that are uncontrolled during evaluation. In this talk, we describe recent work on evaluating path-sensitive error detection techniques that identifies several factors that can significantly influence the cost of analysis. We present results from empirical studies showing that the influence of these factors is sufficiently large that, if left uncontrolled, they may lead researchers to incorrectly conclude that a given technique is cost-effective. While these results illustrate the significant challenges faced in evaluating analysis techniques, they also provide novel insights into the variation in performance of those techniques. We describe how we exploit that variation to significantly speedup the cost of analysis across a range of programs.

Matthew B. Dwyer is the Henson Professor of Software Engineering in the Department of Computer Science and Engineering at the University of Nebraska - Lincoln. Prior to joining UNL, he was a professor in the Department of Computing and Information Sciences at Kansas State University from 1995 through 2004.

He received the BS in Electrical Engineering in 1985 from the University of Rochester and worked for six years as a Senior Engineer with Intermetrics Inc. developing compilers and software for safety-critical embedded systems. His interests led him to pursue his Ph.D. at the University of Massachusetts at Amherst which he earned in 1995.

Dr. Dwyer is an active member of the software engineering, computer-aided verification and program analysis research communities. He has published widely on software specification and static analysis of concurrent programs with an emphasis on developing cost-effective techniques that are usable by practitioners. He is the program co-chair of the ETAPS conference on Fundamental Approaches to Software Engineering (FASE) in 2007 and of the International Conference on Software Engineering (ICSE) in 2008. He also serves as an Associate Editor of the IEEE Transactions on Software Engineering and as Secretary/Treasurer of ACM SIGSOFT.