Benchmarked: Firefox 3.5 beta 4

Firefox

Previously we've told you that Firefox performance on Linux sucks, and, worse, that even Windows Firefox running on Wine is faster than Linux Firefox, so as Firefox 3.5 edges slowly (very slowly!) closer to release, we joined in the Firefox 3.5b4 testday to see how performance is coming along in these later development stages...

We installed the current build of Firefox 3.5b4 ("Shiretoko", or "end of the earth" if you're into minority Japanese languages) on Ubuntu Jaunty and compared it against the stock Firefox 3.0.9 that shipped with the distro by running it through the SunSpider JavaScript benchmark. Below are the results: "FROM" is the original Firefox (3.0) and "TO" is 3.5b4 - what matters is the COMPARISON field, which summarises the performance difference between the two.

Note: this is only a release candidate of beta 4 of Firefox 3.5 (a release candidate of a beta? Eek!) - there's still a lot more testing and development required before release.


TEST COMPARISON FROM TO DETAILS

=============================================================================

** TOTAL **: 1.64x as fast 3616.6ms +/- 5.7% 2206.2ms +/- 7.8% significant

=============================================================================

3d: 1.45x as fast 435.2ms +/- 20.4% 300.6ms +/- 13.2% significant
cube: 1.62x as fast 146.6ms +/- 23.0% 90.6ms +/- 18.9% significant
morph: 1.84x as fast 146.8ms +/- 33.3% 79.6ms +/- 36.8% significant
raytrace: - 141.8ms +/- 30.1% 130.4ms +/- 20.7%

access: 1.74x as fast 533.0ms +/- 12.8% 305.8ms +/- 18.3% significant
binary-trees: ?? 81.6ms +/- 33.7% 100.2ms +/- 39.8% not conclusive: might be *1.23x as slow*
fannkuch: 1.63x as fast 184.4ms +/- 16.3% 112.8ms +/- 27.1% significant
nbody: 2.67x as fast 156.2ms +/- 23.5% 58.4ms +/- 49.3% significant
nsieve: 3.22x as fast 110.8ms +/- 18.7% 34.4ms +/- 41.3% significant

bitops: 3.97x as fast 391.6ms +/- 28.1% 98.6ms +/- 28.3% significant
3bit-bits-in-byte: 17.3x as fast 76.0ms +/- 56.7% 4.4ms +/- 15.5% significant
bits-in-byte: 4.07x as fast 94.4ms +/- 42.2% 23.2ms +/- 30.9% significant
bitwise-and: 14.4x as fast 112.0ms +/- 16.1% 7.8ms +/- 69.8% significant
nsieve-bits: 1.73x as fast 109.2ms +/- 25.8% 63.2ms +/- 43.9% significant

controlflow: ?? 73.4ms +/- 34.9% 76.2ms +/- 43.1% not conclusive: might be *1.04x as slow*
recursive: ?? 73.4ms +/- 34.9% 76.2ms +/- 43.1% not conclusive: might be *1.04x as slow*

crypto: 1.50x as fast 263.6ms +/- 17.5% 176.2ms +/- 16.5% significant
aes: - 106.8ms +/- 31.0% 95.8ms +/- 28.1%
md5: 1.53x as fast 84.6ms +/- 28.2% 55.4ms +/- 23.1% significant
sha1: 2.89x as fast 72.2ms +/- 33.0% 25.0ms +/- 0.0% significant

date: 1.22x as fast 385.8ms +/- 13.4% 317.2ms +/- 5.9% significant
format-tofte: 1.36x as fast 233.6ms +/- 10.8% 171.8ms +/- 14.4% significant
format-xparb: - 152.2ms +/- 25.3% 145.4ms +/- 25.5%

math: 2.18x as fast 357.0ms +/- 21.5% 163.6ms +/- 32.5% significant
cordic: 1.45x as fast 137.2ms +/- 23.9% 94.8ms +/- 26.6% significant
partial-sums: 2.79x as fast 136.6ms +/- 24.5% 49.0ms +/- 58.9% significant
spectral-norm: 4.20x as fast 83.2ms +/- 31.7% 19.8ms +/- 40.3% significant

regexp: 2.31x as fast 293.2ms +/- 4.3% 127.2ms +/- 20.0% significant
dna: 2.31x as fast 293.2ms +/- 4.3% 127.2ms +/- 20.0% significant

string: 1.38x as fast 883.8ms +/- 6.5% 640.8ms +/- 13.3% significant
base64: 2.47x as fast 106.4ms +/- 19.2% 43.0ms +/- 67.4% significant
fasta: 1.54x as fast 202.8ms +/- 19.8% 131.8ms +/- 26.6% significant
tagcloud: - 187.2ms +/- 15.5% 161.6ms +/- 22.7%
unpack-code: 1.25x as fast 264.8ms +/- 12.5% 211.8ms +/- 9.0% significant
validate-input: - 122.6ms +/- 28.8% 92.6ms +/- 31.4%

Why not run the SunSpider benchmark and see for yourself how Firefox 3.5 is doing? The testday is still ongoing and all bug reports are welcome!

You should follow us on Identi.ca or Twitter


Your comments

Better, but not great

These results are not too bad, but could be a LOT better. It shows an improvement, but it's still slower versus browsers like Chrome on Windows.

I would be interested in seeing the latest Windows FF vs Linux FF beta comparison though... :)

What about startup

What about startup time?

Ideally, with clean install, and later with some extensions? :)

Benchmark FF 3.5b4 Vs FF3.0.1 on Linux Debian Squeeze

Very great post, I did my own test on my Linux system and FF3.5b2 is twice as fast, see the results:

TEST COMPARISON FROM TO DETAILS

=============================================================================

** TOTAL **: 2.33x as fast 6421.4ms +/- 5.8% 2751.6ms +/- 4.1% significant

Test the Windows version under Wine on the same Linux box

That's what I'm interested in. It's not surprising FF 3.5 is going to be faster than FF 3.0.x.

What would be surprising is if the FF guys actually started doing the profile-guided optimization on their non-Windows builds.

On Windows Vista Ultimate

On Windows Vista Ultimate SP1 x64 :
** TOTAL **: 2.40x as fast 2304.8ms +/- 1.6% 959.2ms +/- 2.2% significant
FF 3.0.9 (2304.8ms)
FF 3.5b4 (959.2ms)

Crashes on Ubuntu with

Crashes on Ubuntu with segfault ;)

Still slow

Google Chrome continues to blow Firefox out of the water with Javascript performance. And SunSpider is merely a Javascript benchmark... Since most webpages are currently not Javascript heavy, I'm more interested in responsiveness of the UI and startup time and page loading speed when judging whether the performance sucks or not. And there doesn't seem to be any improvement to my eye.

Don't you think that

Don't you think that reducing Firefox's performances to his JavaScript engine performances is a bit meaningless ... ? :--°

Sorry for my english, I'm french =p

@KevinGomez

With Javascript-heavy websites becoming more prevalent every day (Facebook, Gmail), JS performance is by far the biggest bottleneck that a browser has to overcome when giving a good user experience. Facebook in Ubuntu nearly brings my machine to its knees it's such a sluggish experience.

yes.. gmail here in

yes.. gmail here in firefox/linux is way slower than it on windows

Get a real OS: Windows 7

Wow, who ever knew that Linux sucked so bad? The way people talk about it, you'd expect it to be faster than Windows, but its actually significantly slower in every respect. Add to that a dearth of applications, huge stability and security problems, and a community so deep in denial that nothing will ever get them out. How sad.

windows 7 isn't officially

windows 7 isn't officially out yet. Why install an OS that's going to expire and use it as your production OS? What a moron.

RE: Get a real OS: Windows 7

I love a bit of trolling.

Interesting results.
I stopped using FF for a while as it was becoming painfully slow on my Eee-Pc and started using Opera, but I missed the sly old fox and am now back with it.

confirmed over 2x better

in line with what reported by jrchan and Ataa, here's my result: (bear in mind I did the 3.5 beta first, so "as slow" refers to 3.0)
** TOTAL **: *2.34x as slow* 2761.0ms +/- 0.6% 6447.2ms +/- 3.1% significant

Intel Compiler?

You should really do those Linux Test again with the Intel Compiler. It is capable of including several Versions of the Same loop, but optimized for different architecture... of course with those SSE Commands it is obvious the Executables are faster. By repeating over and over again that Linux (as an operating system) is slower, you don't do any good. Of course any Linux Distribution is designed to run on any machine >=i686, and Source Codes are mostly tested with gcc so maybe performance is not the *first* thing they look after.
You (repeatedly) make the impression that Linux (the Kernel) is to blame for the bad Performance.

derdestiller, icc currently

derdestiller, icc currently doesn't manage to actually compile all of Gecko. There's been talk of compiling the JS engine specifically with icc, and doing that does seem to give some speedup over gcc, but introduces the original pain of needing two separate compilers to build the app (and doesn't help most Linux users, who get their Firefox from their distro, in any case).

I'd like to second blizzard's comment: PGO on Linux is blocked on gcc with PGO enabled actually managing to compile the app. Right now it fails to do so.

Of course once it does, the fun moves on to the bugs the profile-guided optimizer introduces. There've been half a dozen such on Windows so far, and I doubt the gcc situation is much better.

Comparision betwenn FF3.5b4 and Midori 0.1.2 (webkit)

TOTAL: 1.47x as fast 9309.4ms +/-10.8% 6338.8ms +/-9.4% significant

Firefox was pwned by Midori. Here is an Athlon 2GHz running Xubuntu 9.04 amd64

BTW, xulrunner 1.9.1 isn't working here.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar