Introducing Lithium, a testcase reduction tool
I wrote a tool called Lithium that automatically reduces large testcases, such as real-world web pages or testcases produced by jsfunfuzz. It can usually reduce a 3000-line jsfunfuzz crash testcase to 3-10 lines in several minutes, considerably faster than I can reduce by hand. Perhaps more importantly, I can do something else while it reduces the testcase.
- Documentation for using Lithium
- Description and analysis of Lithium's algorithm
- Download Lithium (0.9.1)
There are two (related) reasons I'm not calling it "Lithium 1.0" yet. First, I'm hoping to improve the way "interestingness tests" are written. Currently, they're separate programs that communicate to Lithium using their exit code, which limits error handling and might slow Lithium down. I'd like to make the interestingness tests be Python files, but I'm not sure what the best way to do that is. (Should Lithium __import__ the interestingness test? Or should the interestingness test import Lithium and be renamed to e.g. "reduce_crash.py"?)
Second, it would be useful to be able to pass extra arguments to the program being tested. For example, it would be useful to be able to pass a profile name to Firefox, or to pass a Firefox path to Valgrind. One possibility is to put the program being tested last on the command line, so extra positional arguments become options to that program. This solution would only work for interestingness tests that launch a single program (so it wouldn't work for a "renders differently in these two Firefox builds" test, for example), but maybe that's okay. Another possibility is to require the use of a config file for passing arguments to programs being tested (so you don't end up typing all of ".../firefox-bin -P foo" on Lithium's command line).
I'll probably use the MIT license for Lithium (but not for timed_run.py, which was mostly written by Chris Cooper and Bob Clary).
September 15th, 2007 at 2:37 am
Check out the optparse module for parsing options (successor to getopt). It has a lot of flexibility and you could either use an append action for program/args options (meaning each occurrence is appended to a list) or define callbacks to set up your own objects, if necessary.
With the append action these arguments:
./lithium.py -p ../fx2/firefox -a “-P fx2” – p ../trunk/firefox -a “-P fxtrunk”
(as much of a PITA they are to write) would result in a dict like this
{‘p’ : [“../fx2/firefox”, “../trunk/firefox”], ‘a’ : [“-P fx2”, “-P fxtrunk”]}
Of course, using a YAML file or whatever for config, and having each test check for a .cfg file with the same name as the test file is a neat solution too. Avoiding typing is a good thing.
Having each test importing Lithium is probably easier (may require refactoring a bit, wrapping things up in a Lithium object). Although if you want some sort of automation later on, the reverse may be beneficial. Or adapting things in some way to fit into PyUnit or twisted.trial or whatever.
September 21st, 2007 at 11:57 pm
Sounds awesome. I assume though that this only works for bugs where you can programmatically determine if the bug is still present (i.e. crashes, assertions, warnings), as opposed to visual bugs like “this widget is 1px too wide”. Maybe you could hook it up to a fuzzer so that your PC automatically generates crashing testcases and reduces them in an endless cycle while its idle. It could even submit a bug report to bugzilla via email, complete with reduced testcase and breakpad crash id!
I know you came out with jsfunfuzz to fuzz test javascript, but does Mozilla have fuzzers for html, css, xul, etc? I’ve seen and used iexploder, but it doesn’t look like it was mature or actively worked on. It certainly could be expanded.
A while ago, I came out with some javascript bookmarklets that were able to help quickly make a reduced testcase by stripping out parts of a webpage quickly that were unrelated to the bug in question. Would these be of any use now, for bugs where you couldn’t programmatically determine if the bug were still present?
September 22nd, 2007 at 1:46 am
Pretty much. Note that if you can detect the bug from JavaScript, you can have the JavaScript output “FAIL” and have Lithium reduce based on that. For example, you can detect incremental layout bugs from JavaScript if they cause an element’s height to change when it shouldn’t change. I used this trick to reduce bug 368621 with Lithium.
Yes, they would probably still be useful. I bet they work better than using Lithium with an “interestingness test” that waits for you to press ‘y’ or ‘n’ each time :) Where can I find them?
September 22nd, 2007 at 6:36 pm
Let me check if they still work first…
September 28th, 2007 at 5:11 am
OK. I’ve done some clean up, some testing, and fixed a bug, and the result is here: http://www.geocities.com/james_napolitano/TestcaseBookmarklets.html. This was my javascript-learning exercise from 3 years ago. I used many of your bookmarklets as a model. Let me know what you think.