Indistinguishable from Jesse

I gave my new fuzzer a break from testing TraceMonkey by asking it to look for differences between SpiderMonkey and JavaScriptCore. I have listed them below, with SpiderMonkey output above JavaScriptCore output.

I have no idea how many of these are bugs (in SpiderMonkey or JavaScriptCore) and how many are ambiguous in the spec (intentionally or unintentionally).

Early error reporting

SpiderMonkey reports some errors at compile time that JavaScriptCore only reports at run time, if the code is actually hit. The difference is most obvious (and most likely to cause compatibility problems) if the code is skipped.

> if (false) { --1; }
S: SyntaxError: invalid decrement operand
J: (no error)

> if (false) { return; }
S: SyntaxError: return not in function
J: (no error)

instanceof

The two engines disagree about what objects are reasonable operands for the 'instanceof' operator.

> ({} instanceof {a:2})
S: typein:3: TypeError: invalid 'instanceof' operand ({a:2})
J: false

> ({} instanceof eval)
S: false
J: Exception: TypeError: instanceof called on an object with an invalid prototype property.

new with native functions

SpiderMonkey allows the "new" operator to be used with some native functions that JavaScriptCore considers non-constructors.

> new Math.sqrt(16)
S: 4
J: Exception: TypeError: Result of expression 'Math.sqrt' ... is not a constructor.

> new ({}.toString)
S: [object Object]
J: Exception: TypeError: Result of expression '({}.toString)' ... is not a constructor.

> new eval
S: typein:9: EvalError: function eval must be called directly, and not by way of a function of another name
J: Exception: TypeError: Result of expression 'eval' ... is not a constructor.

Converting between numbers and strings

> print(+'\00000027')
S: NaN
J: 0

> (1e-10).toString(16)
S: 0.000000006df37f675ef6ec
J: 0

const

There are subtle differences in handling of this new keyword.

> const d; const d;
S: TypeError: redeclaration of const d
J: (no error)

> const c = 0; print(++c);
S: 0
J: 1

Other differences

> print((function(){return arguments;})());
S: [object Object]
J: [object Arguments]

> typeof /x/
S: object
J: function

See Mozilla bug 61911, which changed this in SpiderMonkey in 2007.

Making JavaScript faster is important for the future of computer security. Faster scripts will allow computationally intensive applications to move to the Web. As messy as the Web's security model is, it beats the most popular alternative, which is to give hundreds of native applications access to your files. Faster scripts will also allow large parts of Firefox to be written in JavaScript, a memory-safe programming language, rather than C++, a dangerous footgun.

Mozilla's ambitious TraceMonkey project adds a just-in-time compiler to Firefox's JavaScript engine, making many scripts 3 to 30 times faster. TraceMonkey takes a non-traditional approach to JIT compilation: instead of compiling a function at a time, it compiles only a path (such as the body of a loop) at a time. This makes it possible to optimize the native code based on the actual type of each variable, which is important for dynamic languages like JavaScript.

My existing JavaScript fuzzer, jsfunfuzz, found a decent number of crash and assertion bugs in early versions of TraceMonkey. I made several changes to jsfunfuzz to help it generate code to test the JIT infrastructure heavily. For example, it now generates mixed-type arrays in order to test how the JIT deals with unexpected type changes.

Andreas Gal commented that each fuzz-generated testcase saved him nearly a day of debugging: otherwise, he'd probably have to tease a testcase out of a misbehaving complex web page. Encouraged by his comment, I looked for additional ways to help the TraceMonkey team.

JIT correctness

Differential testing is designed to find correctness bugs. It runs a randomly-generated script twice (with and without the JIT) and complains if the output is different.

It quickly found 13 bugs where the JIT caused JavaScript code to produce incorrect results. These bugs range from obvious to obscure to evil.

It even found at least one security bug that jsfunfuzz had missed. An uninitialized-memory-read bug caused output to be random when it should have been consistent. jsfunfuzz missed the bug because it ignores most output, but the differential testing caught it just like it would catch a JIT vs interpreter difference.

JIT speed

I set up the new fuzzer to compare the time needed to execute scripts and complain whenever enabling the JIT made a script run more slowly. It measures speed by letting the script run for 500ms and reporting the number of loop iterations completed in that time.

So far, it has found 4 serious bugs where the JIT makes scripts several times slower. Two of these have already been fixed, but the other two may be difficult to fix.

It has also found 10 cases where the JIT makes scripts about 10% slower. Most of these minor slowdowns are due to "trace aborts", where a piece of JavaScript is not converted to native code and stays in the interpreter. Some trace aborts are due to bugs, while others are design decisions or cases for which conversion to native code simply hasn't been implemented yet.

There is some disagreement over which trace aborts are most likely to affect real web pages. I asked members of Mozilla's QA team to scan the web in a way that can answer this question.

Interpreter speed

Mostly for fun, I also looked to see which code the JIT speeds up the most. Here's a simplified version of its answer:

for (var i = 0; i < 0x02000000; ++i) {
  d = 0x55555555;
  d++; d++; d++; d++; d++;
}

This code runs 250 times faster when the JIT is enabled. The JIT is able to achieve this gigantic speedup due to the interpreter being inefficient in dealing with undeclared variables and numbers that can't be represented as 30-bit ints.

Assertions

The JavaScript engine team has documented many of their assumptions as assertions in the code. Many of these assertions make it easier to spot dangerous bugs, because the script generated by the fuzzer doesn't have to be clever enough to actually cause a crash, only strange enough to violate an assumption. This is similar to my experience with other parts of Gecko that use assertions well.

Other JavaScript engine assertions make it easier to find severe performance bugs. Without these assertions, I'd only find these bugs when I measure speed directly, which requires drastically slowing down the tests.

More ideas

One testcase generated by my fuzzer demonstrated a combination of a JIT performance bug with a minor bytecode generation bug. I might be able to search for similar bytecode generation bugs the same way I searched for decompiler bugs: by ensuring that a function does not change when round-tripping through the decompiler. In order to do that, I'll need a new patch for making dis() return the disassembly instead of printing it.

I should be able to find some performance bugs by looking at which aborts and side exits are taken. This strategy would make some performance bugs (such as repeatedly taking a side exit) easier to spot.

I updated Tidybox today. The most requested change I made: fix it to work on static Tinderbox pages, such as http://tinderbox.mozilla.org/Firefox, in addition to the showbuilds.cgi URLs.

Install the new version

I created a page called Reducing testcases to replace the old Gecko BugAThon page. Now I have something modern to point bug reporters at when I'm too lazy to make a reduced testcase myself :)

The page is on the developer.mozilla.org wiki, so you're welcome to help improve it.

I made this using the On Notice Board Generator.

Despite all the problems, the summit in Whistler has been worth it:

• I finally got to meet Mozilla security contributor Paul Nickerson. He had interesting ideas about how to find and prevent JavaScript privilege escalation bugs, which we discussed with Blake Kaplan.
• Firebug developer John J. Barton and I figured out why Firebug showed user agent stylesheet rules on his machine but not mine. He fixed the bug on the spot. This should let me switch from the aging DOM Inspector to Firebug for reducing layout testcases, saving me time.
• Rob Arnold took one of my extra Apple remotes and said he'd try to get it working with Firefox on Windows.
• Dave Mandelin's session about Static Analysis was enlightening.

I avoid Keynote and PowerPoint to ensure that I will be able to refer to my presentations three years from now. Until last week, I thought that using the HTML-based S5 format meant compromising the style of my presentations.

While preparing a presentation for the Mozilla summit, I found a pretty S5 theme called Glossdeck. More importantly, I learned that Opera supports the Apple Remote for switching between slides.

I'd love to be able to do the same using Firefox rather than Opera. Using Opera is somewhat painful because switching slides using the keyboard is awkward (it uses space and shift+space rather than arrow keys) and Glossdeck has to be modified to work in Opera. But Firefox on Mac doesn't have a full-screen mode or support for the Apple Remote.

Boris Zbarsky's work in bug 113934 for reparenting Firefox tabs between windows should pave the way for Firefox to have full screen on Mac. It would be great if Firefox had Apple Remote support as well, perhaps as a DOM event sent to the focused web page.

If anyone at the Mozilla summit wants an Apple Remote, I brought three extras. I hear they work with Keynote and PowerPoint too.

In 2006, Mike Beltzner filed a bug saying that Firefox's about:config should have a warning. Chris Thomas wrote a patch adding a warning page, and it was checked in with a playful title suggested by the same Mike Beltzner: "Be careful, this gun is loaded!".

Some people thought the reference to guns made Firefox too violent. After much discussion, Beltzner changed the title to "This might void your warranty!", which was a suggestion from Phil Ringnalda.

Today, Christopher Aillon of Red Hat filed a bug about the "warranty" string. He says it has caused several users to contact legal departments or IT departments with questions that should have been unnecessary.

My suggestion is "Caution: Firefox internals may be hot". As a bonus, it fails to make sense in Iceweasel-branded versions.

Additional suggestions may be hidden in the Firefox source tree. When Beltzner made the change from "gun" to "warranty", he also added a note to localizers, suggesting that the title need not be a direct translation from English but "should be attention grabbing and playful". At least three localizers substituted their own phrases. I'm curious what the strings say when translated back into English.

Firefox 3 added support a new CSS color keyword, transparent. Surprisingly, this broke some sites, many of which had rules like table { color: transparent; } due to a Microsoft FrontPage bug.

The strangest part: Firefox wasn't the first major browser to support transparent. Safari was.

These sites were broken in Safari too — until the webmasters got emails from Firefox users. Is Safari's market share really so low that even when Safari is the first to make a change that affects compatibility, Firefox helps Safari more than Safari helps Firefox?