Fuzzing for JavaScript correctness
Fuzz-testing is usually only used to find crashes and assertion failures, but my JavaScript engine fuzzer goes beyond catastrophic failures when it tests the decompiler. It checks the decompiled code for signs of incorrectness in two ways.
First, it checks that the decompiled code compiles without giving syntax errors. This finds fun bugs like bug 346904 where the decompiler screwed up in an understandable way, as well as bugs like bug 351496 where the decompilation is complete nonsense.
Second, it checks that the decompiled code is canonical -- compiling and decompiling again should give the exact same representation as the original decompilation. This helps find bugs like bug 381196 where decompilation changes the meaning of the code without introducing a syntax error.
Some decompilation changes, such as bug 352068, did not change the meaning of the code and simply reflected varying amounts of optimization in the compiler. Early in the fuzzer's life, I was able to convince Brendan that it was worth fixing many of those otherwise harmless "round-trip changes" in order to make it possible to find other bugs with this method.
This pair of checks doesn't find all decompiler bugs, of course, but it finds quite a few of them. jsfunfuzz has a few other correctness checks for things like unnecessary parentheses in decompiled code and bogus results from object uneval.
Can you think of other ways to use fuzz-testing to find "correctness" bugs?
August 3rd, 2007 at 1:47 am
I’d been thinking about fuzzers for Mozilla-the-platform but wasn’t aware of your work. My thoughts were geared more towards an XPCOM fuzzer, after reading about ActiveX/COM fuzzers finding heaps of bugs in IE. (It seems like a logical vector, COM being what it is.)
Generally, I think it would be very beneficial to bulk-test all XPCOM interfaces with weird input data to catch any systemic or local problems arising from that. It needn’t be that they’re accessible from the web or that they are part of privilege escalation, but interface correctness and avoiding “garbage in — crash-go-boom out” I suppose is always a high priority. (I’m unsure how much testing goes into XPCOM interfaces to begin with, it may be redundant to call in the fuzz.)
August 3rd, 2007 at 2:13 am
I think it’s reasonable for an object’s code to crash if the object is given input that’s inconsistent with invariants stated in its interface. There are some “internal” objects that would be great to fuzz, but just because something is an XPCOM object doesn’t mean it needs to be paranoid.
August 3rd, 2007 at 2:43 am
Of course. Though “weird” in my mind doesn’t necessarily mean “out of domain”, Carrot Top is still (well, arguably) a human being.
The goal would naturally be to try and isolate cases where a crash is highly unpredictable and means that it could either go boom, elevate privileges or cook you a steak dinner. From what I read about the ActiveX fuzzer (I’ve misplaced the link, there was something about it on SecurityFocus, though) it found some really nasty things no one expected. In the scope of these things, that would be a really good thing.
August 3rd, 2007 at 4:13 pm
Not sure if this is what you are looking for – but hinky code designed to break on being broken could be fun.
http://isc.sans.org/diary.html?storyid=3219
“After downloading the JavaScript file it was obvious that all function and variable names are complete random. Further to that, the deobfuscation function used the well known arguments.callee.toString() trick in order to prevent modification of the code (so you just can’t replace the final document.write() call to something else as this will break the deobfuscation function body – attempts such as this one typically throw the function into an endless loop).”
I am sure the people at isc.sans.org would send you some samples to throw in. Asking for stuff that has been designed not to be compiled might flush out a few corner cases and raise the temp in the arms war a bit more.