Browser benchmarks 2: even Wine beats Linux Firefox

Mozilla

We posted yesterday about Firefox having very different JavaScript performance on Windows and Linux, despite being the same version of the software.

Some people have said that we should have used a stock build from Mozilla. (We disagree, because we'd argue that most Linux users use software from their package manager rather than downloading bits and pieces from the web.) Others have said that Opera should be tested. And some people have said that it's Nvidia/AMD/Intel drivers that are slowing down Linux.

Anyway, we thought we would conduct a couple more quick benchmarks to see whether we can eliminate some of these variants. We don't have time to run the full benchmark suite and fiddle around from scratch, so we ran just a few quick tests to see what we could find.

This time the information you need to know is:

  • These benchmarks were run on the same computer as before, running the same Fedora 10 install.
  • We tried Mozilla Firefox for Linux as downloaded straight from Mozilla.
  • We also tried Mozilla Firefox for Windows as downloaded from Mozilla, but running it using Wine on Fedora.
  • We installed and tested Opera 9.63 for Fedora 10, as downloaded from Opera.com. Note: we were only able to find i386 builds on the Opera site; this isn't optimal so if someone can point us towards an i686 build for Fedora 10 we will happily update the article.
  • We ran the Google V8 Benchmark suite V3, as before.

To be absolutely clear: we took the Windows Firefox and ran it on Fedora Linux using Wine 1.1.12 as provided by Fedora:

  • "Firefox Windows" is Firefox running on Windows.
  • "Firefox Fedora" is Firefox running on Fedora using the Fedora package.
  • "Firefox Mozilla" is Firefox running on Fedora using the Mozilla build.
  • "Firefox Wine" is Firefox as compiled for Windows running on Fedora using Wine.
  • "Opera" is, well, Opera 9.63 running on Fedora.

With all that in mind, here's how the results look now:

Firefox on Windows, Linux and Linux/Wine

Firefox running on Windows, Linux and Linux/Wine, plus Opera.

The end result: Firefox from Mozilla or from Fedora has almost nil speed difference, and Firefox running on Wine is faster than native Firefox. Opera lags behind, but we're inclined to believe that number would increase if a better build was used.

Is this the end of the story? Probably not - we expect some commenters will come up with some other reason why the slowdown is clearly Nvidia/ATI/someone else's fault. Go on, folks: try the benchmark yourself. If you're running Linux, install Wine and try Windows Firefox on Wine and make up your own mind.

You should follow us on Identi.ca or Twitter


Your comments

Jithin

I dont know about the bench marks. But from my use experience I always felt Opera is way more fast that firefox from my ubuntu repository.

Ubuntu?

Add Ubuntu to your benchmarks!!!

Optimizing compilers?

<i>"To the poster who said that "There is only so many ways you can write an optimizing compiler". It is exactly the opposite. It is actually a fun fact - it is proven that there are infinite number of ways to write an optimizing compiler."</i>

Kudos to the first person who can write a compiler that optimizes your app to 0kb!

Midori?!

Firefox Linux: 153
Epiphany: 155
Midori (WebKit): 1473 (what the hell?!)

I confirm this too

What was an unspoken truth has come out as a big shout!

Firefox on Linux needs optimization and the developers should give back to the community from which they derive much.

Firefox on windows is a lot faster and I usually find it faster while running my Firefox Portable using wine.

I think this is due to Firefox developers giving more time to their bigger customer slice who are on Windows. __After all why give a damn to Linux users who have limited choice in case of web browsers (we don't have IE or Chrome and Opera or Konqueror are not much choice).__ (pun intended)

Strange this happens with other apps too!

Sumatra PDF again downloaded from portableapps.com, renders pdfs faster than native OKular or GNOME's Evince

This is pretty sad...

This is a pretty sad article on the part of the author. In fact it makes me question the technical competence of the author.

In your test the Linux system is running both Wine and FF. FF is running native code and FF-Wine is running an "emulation" layer thats still running native code under it. FF-Wine has an additional layer of work to do. So if Linux itself is slow then its going to be slow in both cases. The only differences between the two are the FF code bases (Win vs Lin) or the graphical toolkits used (GTK, QT).

Now maybe someone can come up with a way to test out GTK and QT so that we can eliminate them from the equation. However I get the feeling that the author doesn't really care about the suggestions. He just wants to make a case that Linux is slow without trying to find the reasons and take them to the devs.

Also...

Have you tried comparing other browsers on both systems to see if this is a Firefox phenomenon or if Linux runs browsers slower across the board. I just tried Midori for comparison and if beat the breaks off FF on Ubuntu.

You don't seem to be trying to eliminate any variables here. You just run one browser on two OSes and proclaim one to be slower. Do you really want to get to the root of the problem?

Now you can whine and scream about the average user not using Midori but if you do then you have missed the point ENTIRELY. If you're a Linux fan and want to make it better then lets do some real investigative work and track down the problem.

My tests

Gentoo on 1.6Ghz Pentium M, GCC 4.1.2
CFLAGS="-Os -march=pentium-m -pipe -fomit-frame-pointer"

on google v8 test:
native Firefox 3.0.6 129
wine Firefox 3.0.6 150

Debian Bechmarks

Debian sid 64-bit. Tests were performed 5 times. Error is standard deviation.

iceweasel amd64 3.0.6-1: 140 +/- 3
iceweasel i386 3.0.6-1: 123 +/- 3
firefox i686 3.0.6 : 118 +/- 2
firefox wine 3.0.6 : 144 +/- 4

It looks like we have a problem.

Conclusion

It is clear that the companies prefer to spend most of their time fine tuning their strategic and time consuming apps for the leading OSes. Period.
It is quite reasonable spending almost no time for supporting an OS that has less than 3% market share, according to other stats - less than 1%.

For now, if the Linux community wants better code, it must either greater market share or participate itself and improve the quality of the corresponding portions of the Linux code.

Things that it is NOT due to

No the Linux kernel, X, GCC or graphics drivers. These are all involved in Firefox for Windows running under Wine.

So it is down to two possibilities:

(1) The GUI toolkit/widgets used in Firefox native Linux, or
(2) Firefox native Linux core code itself.

I'd opt for (2) above, given that the Windows version of Firefox uses an optimisation technique call "Performance Guided Optimisation" (PGO), where the Linux version currently does not.

Apparently this is soon to be fixed, however:

"The windows version of Firefox has got PGO enabled since version 3. The Linux version does not. Version 3.1 WILL be compiled with PGO enabled for Linux."

So, if this lack of optimisation in the compilation of the Linux version of Firefox for versions earlier than 3.0.x is indeed the culprit, then happily the next release of Firefox for Linux should see this performance difference disappear.

JAVASCRIPT PROFILE GUIDED OPTIMIZATION

The explanation for this is javascript Profile Guided Optimization for JIT features. This and Javascript was introduced into Firefox and Chrome to boost javascript performance by a factor of 10 to 20 over existing browsers like IE6. PGO has been incorporated into Windows Firefox in version 3.0x whereas it will be introduced into Linux versions of Firefox in version 3.1. It is also the explanation why IE is massively slower than the Linux versions of Firefox on the same benchmark (by a factor of more than 3) - IE6 does not have any javascript JIT features at all.

Basically, the bottom line is that the Firefox Linux version is a couple of months behind the Windows Firefox in the implementation, but implemented it will be. Unfortunately as can be seen from a number of the comments, this has not stopped wishful thinking Windows fudsters, fanboys, shills etc. proclaiming that Linux is a toy, what would you expect from a free OS, it shows Linux is obsolete etc. Just part of the course I guess.

V8 Benchmarks aren't about rendering...

@the anonimous author of "The beginning of the end"
THANKS you made me laugh :) please keep trolling about unknown

btw. the V8 benchmark suite isn't about rendering, it's a test to know how much time the javascript engine takes to execute an algorithm, as stated by google's page: "This page contains a suite of pure JavaScript benchmarks that we have used to tune V8."

So, GTK and X here are not a problem (you are not rending anything), Linux (i'm talking about the kernel) is not really touched. The poor performance is caused by the less-optimized javascript engine in firefox's unix code (even in OSX firefox score sucks)

The benchmark's purpose is to give a score to the Javascript (ECMA not java) engine.

peace:)

Other benchmarks suites?

I have not looked too closely at what the google benchmarks actually do to come up with these scores. And even with everything else perfectly aligned, the performance difference between ff on win vs lin is probably still a mystery.

But I can't help wondering whether the 300% speed increase going from ff to webkit could be a feature of the benchmark suite itself, rather than a compiler/code-quality thing? And if that is the case, perhaps something in the windows build is somehow treated preferential in the benchmarks as well?

Is there another benchmark suite out there, besides the google suite, that we could try to verify these results with?

FWIW, I get 193 in Firefox under Fedora 10 (x86_64) and 807 using webkit on the exact same machine.

One more interesting data point

It would be interesting to see the same benchmark run under the Linux version 2.0pre3-alpha Seamonkey. Some of the operations are notably faster. And for grins including Chrome would give people a reference point.

Opera results

My scores

Opera 9.63 static-qt (Debian 5.0): 91
Opera 9.63 Wine (Debian 5.0) 165

Disturbing.

Google Chrome 1.0

Please show scores os google chrome it's twice as fast as ff on windows ;)

It's gotta be a Microsoft

It's gotta be a Microsoft fault somehow. Maybe Mozilla is being paid by M$ to make Firefox slower on Linux.

FreeBSD results?

An easy way to see if GCC is responsible would be to compile FF using FreeBSD's compiler - IF it is better, than we have a possible cause, but if it's the same, the result is also pretty much useless...

Interesting the choices made in this test.

>Javascript used to be interpreted until very recently Google Chrome, Firefox, and Safari introduced JIT compiling to increase javascript speed by a massive factor, however all the features are not enabled yet on all browser variants.

>On the Webkit benchmarks the two slowest benchmarks by a large margin are IE7 and IE8.

>PGO isn't enabled yet on the Linux version of Firefox, and hence it currently runs slower than Firefox on Windows, but much faster than IE7 or IE8 on Windows. PGO will be enabled soon on Firefox Linux versions and then Firefox will run as fgast or faster on Linux than on Windows.

In other words, almost any comparison of javascript performance on Linux versus Windows would have yeilded the result that the Linux version was much faster ... except for the versions chosen by the original authors.

In order to get the result where Windows javascript performance was better, the original posters HAD to choose Firefox 3.0.x on Windows XP versus Firefox 3.0.x. on Linux.

ANY other choice for comparison would have yeilded opposite results. Choosing to compare Linux vs Vista, or choosing Firefox 3.1, or choosing any version of Firefox 2.x, or choosing to compare Firefox on Linux against IE (any version) on Windows (any version) ... would have resulted in the result that Firefox on Linux had the better javascript performance.

Interesting.

Agenda, perhaps?

I am not sure there is an agenda...

...at least with regard to the author of the article. Firefox 3.1 hasn't been released you except as a beta. Some of the Windows fanboy/shill posters clearly seem to have an agenda which they are very quick to jump on though.

The single most sensible suggestion

As posted recently, you should indeed compile the windows version with gcc and recompile the linux version with gcc using the same options and see if indeed the compiler is implicated.

Having established whether (or not - and my gut feel is not) this is the case, then all that remains is poorly optimised Linux code within Firefox. Profiling ought to pin that down, I would have thought. I remember years ago having a performance issue with a Windows application and finding that the leading bottleneck was a call to get the decimal point character for the locale - the developer had called this virtually every time a number needed formatting and it turned out to be a surprisingly expensive call.

GCC

It is generally known that GCC does not produce FAST code.

It may be good, it may be free, but the result is not FAST.

Try compiling a program with compilers that does not use GCC as backend, like freepascal, and you will be surprised of executions almost instantaneous... which is normal with the super fast CPUS we are using this days.

To be absolutely clear

As we can see in the graph Firefox Mozilla got 181 bananas in the benchmark whereas Firefox Wine got 227 druken penguins.

Is it so hard writting WTF the units mean!? comparison with a reference system (as in this case), DOM accesses per second, time to load whatever... And at least saying something like "higher is better".

On my rig, and the numbers

On my rig, and the numbers are not terribly important, Epiphany outperforms Firefox on both the Google V8 v3 and Sunspider.

GTK?

Isn't it obvious that it's GTK that's slow here?

robert@ocallahan.org

Our Windows build uses profile-guided optimization and our Linux build does not. Also, MSVC generally produces faster, smaller code than gcc. That is easily enough to explain the differences you're seeing here. It also explains why running the Windows build under WINE is faster.

Almost all Linux Firefox users are actually using a distro packaged build. So hassle your distro to use PGO if they aren't already.

Also, be aware that the Google V8 benchmark is fairly narrow and bogus; it basically tests recursive method calls. Sunspider is a much better all-round benchmark.

robert@ocallahan.org

Oh, just to allay conspiracy theorists: far more Mozilla developers use Linux than Windows. Partly because it takes about 20 minutes to build Firefox on a good Linux box vs an hour on Windows. And also of course because of free software.

Windows still rules

Just wait till Windows 7 comes out. I exhaustively benchmarked the latest IE8 beta and it is two orders of magnitude faster than anything else on the market, AND I did it on my netbook with only 512 mb of ram. Web surfing on Windows 7 compared to Linux and OS X is like comparing an F1 race car to a horse drawn cart or a 60's Volkswagen Beetle, respectively.

.

.

GCC slow? No it isn't.

I have a project at work that looks up items in a custom memory mapped database. It was developed in C++ on Linux and ported to Windows, built on Server 2008 with VC 2008.

It does a lookup about 5 times slower on Windows built with MSVC. It is all in the compiler too, because the inner code uses no API calls at all.

There are a couple of reasons for that speed difference. The Linux version does profile optimization and Windows doesn't, because of bugs using certain types of DLLs for profiling in VC2008.

The GCC version gains a lot of speed by mashing all the source files together into single sources as an alternative to the IPO that MSVC offers.

That probably isn't practical for Firefox. The resulting pile of source would be massive and GCC would probably want 20 GB of RAM to compile it. But I believe it does show that GCC does not produce slow code.

Swiftfox is not the solution and FF 3.1b2 does worse

I'm using Kubuntu 8.10, with native FF 3.0.6 from the repos I get 161 and with Swiftfox I get only 171, not good enough IMHO, On WINE I get around 188.

Ironically FF 3.1b2 which uses the new TraceMonkey JS engine gets 154 when using the native linux binary and 173 when using WINE(javascript.options.jit.content = true in both)

My thoughts

The main reason for Firefox-Windows performing faster than Firefox-Linux is the default GCC compiler flags used in the builds:

Fore Windows Mozilla defaults to -O2 optimization;
for Linux Mozilla defaults to -Os optimization.

This is apparently because of the huge file size reduction offered by -Os on Linux.

I have compiled Firefox for Linux using the -O3 flags and the performance is dramatically increased on my Athlon box.

Midori 2

On same hardware both using libgtk-x11-2.0 :

FF : Score : 71 ->
Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.9.0.6) Gecko/2009020407 Iceweasel/3.0.6 (Zenwalk GNU Linux)

Midori : Score: 775 ->
Midori 0.1.1-a4e6720

x10 !!!! Mozilla team : how about checking your code ?

I am GAY,OK?

I am GAY,OK?

Maybe it's due to the

Maybe it's due to the wine-gecko implementation

Upside down graph anyone?

It seems to me... that opera is winning, native firefox is in second, and windows based ff is last. Why? Because its up to my interpretation. You see, the original poster obviously failed his/her graph theory course, as there are NO metrics whatsoever on that benchmark.

Is that iterations per second? Some vague qualitative analysis not based on any real world metric? If its the latter, this means diddly. If its the former, you're all ranting on the wrong build.

Just thought I'd point that out, since 25 points here and there could mean as much as a split second for all we know.

No surprise

I used linux for a year now, and I'm quite aware of the difference. Occasionally I'll launch Opera or Firefox in Wine to get a page that freezes in linux. To me it's rather a disappointment, however I didn't leave windows for the performance issues - more because I really wanted to get away from the kind of evil inherent to the platform.

V8 benchmark is probably not the best.

Hi guys, just thought I'd let you know that I've repeated the test with the latest 3.1 candidates and got similar results with the V8 benchmark, Wine Firefox comes in at 118 on my machine while Linux Firefox scores only 61.

However on the SunSpider benchmark, Firefox Wine and Firefox Native are *much* closer, with Firefox Wine coming in at ~2500ms and Firefox Native at ~2800ms.

Amazingly on Celtic Kane's Javascript test Native Firefox is shown as being over twice as fast as Wine Firefox, coming in at ~530ms vs ~1100ms.

So it seems that this whole thing could be much ado about nothing as Shakespeare said, and serves as a lesson about why we should take micro-benchmark results with a grain of salt.

Unless, that is - that you believe Firefox on Linux can be simultaneously half as fast, just as fast and twice as fast as it's Windows cousin.

However, after saying all that, I was unable to test Firefox in Windows on this machine, so I assume there is still going to be a fair speed deficit between Firefox on Windows and Firefox on Linux... at least as far as microbenchmarks are concerned.

Why do you wonder about that

I am using both, Windows and Linux (at work) and I really hate using Linux as a Desktop because it is just way slower than Windows is.
It feels like working with a 386 machine especially when I use Firefox.
I wonder why there aren't more people complaining about that. And please don't think that I am an unexperienced user who is not able to configure his system properly. I am using Linux and Windows for ages.
Every kind of browser is working in windows much better compared to Linux and I am not talking about the rendering time. No, its about scrolling, switching tabs etc. Linux just feels (and it is) way slower. Don't misunderstand me, I really like Linux and I am not a Microsoft fan at all but we just need to realize that Linux Desktops are far behind Windows ones.

Cheers,

Fedora user :-)

Wow, that's a lot of data in

Wow, that's a lot of data in the comments. Maybe it coul be leveraged to get some meaningful statistics (kind of a poor man's Phoronix Test Suite for browsers... and that would be a really poor man).

Anyway, here's my results, running KDE4 with openSuSE 11.1 on a Celeron M 1.6 with 1.5 GB RAM and a homebrew 2.6.28.7 kernel:

Firefox - Linux (3.0.7): 87,8
Firefox - Wine (3.0.7): 95,2
and, surprisingly,
Minefield - Wine (3.1b2): 93,2

Admittedly, I do have quite a few add-ons installed on my linux build that I didn't care to install in Wine. But, all in all, gosh, are those results low!

Ubuntu 8.10 my results on

Ubuntu 8.10

my results on Sunspider

firefox 3.0.7 native => 6065.0ms +/- 5.8%
minefield 3.2.a1pre native => 3361.8ms +/- 17.3%
firefox 3.0.7 under WINE => 4926.4ms +/- 2.8%

V8 Benchmark Suite - version 3

firefox 3.0.7 native => Score: 112
minefield 3.2.a1pre native => Score: 147
firefox 3.0.7 under WINE => Score: 155

hope it helps, does anyone know in Minefield is using PGO??

Compiler.... compiler

You did not compare apples to apples, but also means that we need to be fairly reasonable.

The compiler used in Firefox Winodws build have a bigger inherited level of optimizations. The same issue appears also in OpenOffice.

Anyway, your machine is a 64 bit capable, and for the sake of comparison, the 64 bit binaries in computational intensive tasks gain between 10-20% in real world applications. So you may get the same improvement using the 64 bit version of the distros. In real life there is no reason why you shouldn't use a 64 bit version. Also, the Firefox builds are 32 builds, so they will not give any optimizations.

What should be done in future to not make the difference so big? Mostly the distro makers should use Profile Guided Optimizations using the most used sites: gmail, ymail, and JS benchmarks. This will get a win of 5% overall (it may be as 15% in 2-3 cases) but will reduce the gap. As most Distro makers do not do that, if I would have your machine and I want Linux and speed, and no head hassle, for sure I would pick a Linux 64 distro!

another benchmark

hey there, i did my tests too

Details : FF 3.0.8, Fedora 10 x86_64, 2.6.27.21
AMD Athlon(tm) 64 Processor 3500+
cpu MHz : 2200.000
(during tests)

Results :

Score: 74.5
Richards: 36.9
DeltaBlue: 75.9
Crypto: 84.1
RayTrace: 98.0
EarleyBoyer: 114
RegExp: 65.2

What?

How the heck does this happen? Shouldn't it take slower since WINE has to load libraries and whatnot? That is really weird.

Actually, on my computer, it takes windows about 5 seconds to load Firefox, while on Linux, it takes about 2.

What?

How the heck does this happen? Shouldn't it take slower since WINE has to load libraries and whatnot? That is really weird.

Actually, on my computer, it takes windows about 5 seconds to load Firefox, while on Linux, it takes about 2.

Whoops

Double post

How long have you been playing ball?

Compiler optimizations will yield better performance. Glibc, GCC, and the Linux kernel compiled for the architecture will perform better.

* note: architecture meaning i486, i586, i686 and Pentium 4, Pentium Core Duo, Pentium Core 2, Xeon

Windows probably has binary checks to enable features for different architectures. Only Windows operating system developers from Microsoft could truly explain this.

With all that said,

Fedora enables SELINUX by default. This is more of a hinderence to performance. Their kernel is compiled Pentium 2, i586, probably with the server preemption kernel option and 250 hz timer resolution option.

Ubuntu:
A better distribution to test against because they are user oriented supposedly. Perform your tests again on Ubuntu 8.04 which is the current maintained distribution of Ubuntu. I'd advise avoiding Ubuntu 9.04 as there are major issues with the Intel graphics drivers. This will reflect what the majority of users currently use.

Identifying the problem:
So the real culprit here is the kernel and operating system.
If you are seeking the results you get from wine and Windows then recompile the kernel with the following options.
* CPU type, ie. Core 2 or whatever your cpu architecture is
* 1000 hz timer
* Voluntary Preemption
* Tickless Timer Optional
* Disable kernel profiling

Recompile GLibC for i686 cpu architecture.

Recompile Xorg for i686 cpu architecture.

Recompile Firefox for i686 cpu architecture.

Disable Xorg's compositing.

Run your test again and be amazed.

ArchLinux and Gentoo handle the majority of what I've outlined here. They are superior versions to all other Linux distributions when seeking performance.

Call-outs:
Nothing is ever handled upstream. This is a bad practice. If you're going to help save the world .... computers, come on people need food, water, and good teachers. Machines will be the death of the human race.

Thank you for your time.

Epiphany wins no competition

These is the score of epiphany:

Google V8 Benchmark suite V3: 2898

Epiphany Web Browser 2.27.2
Dell XPS - ArchLinux 64bit

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar