Browser benchmarks 2: even Wine beats Linux Firefox

Mozilla

We posted yesterday about Firefox having very different JavaScript performance on Windows and Linux, despite being the same version of the software.

Some people have said that we should have used a stock build from Mozilla. (We disagree, because we'd argue that most Linux users use software from their package manager rather than downloading bits and pieces from the web.) Others have said that Opera should be tested. And some people have said that it's Nvidia/AMD/Intel drivers that are slowing down Linux.

Anyway, we thought we would conduct a couple more quick benchmarks to see whether we can eliminate some of these variants. We don't have time to run the full benchmark suite and fiddle around from scratch, so we ran just a few quick tests to see what we could find.

This time the information you need to know is:

  • These benchmarks were run on the same computer as before, running the same Fedora 10 install.
  • We tried Mozilla Firefox for Linux as downloaded straight from Mozilla.
  • We also tried Mozilla Firefox for Windows as downloaded from Mozilla, but running it using Wine on Fedora.
  • We installed and tested Opera 9.63 for Fedora 10, as downloaded from Opera.com. Note: we were only able to find i386 builds on the Opera site; this isn't optimal so if someone can point us towards an i686 build for Fedora 10 we will happily update the article.
  • We ran the Google V8 Benchmark suite V3, as before.

To be absolutely clear: we took the Windows Firefox and ran it on Fedora Linux using Wine 1.1.12 as provided by Fedora:

  • "Firefox Windows" is Firefox running on Windows.
  • "Firefox Fedora" is Firefox running on Fedora using the Fedora package.
  • "Firefox Mozilla" is Firefox running on Fedora using the Mozilla build.
  • "Firefox Wine" is Firefox as compiled for Windows running on Fedora using Wine.
  • "Opera" is, well, Opera 9.63 running on Fedora.

With all that in mind, here's how the results look now:

Firefox on Windows, Linux and Linux/Wine

Firefox running on Windows, Linux and Linux/Wine, plus Opera.

The end result: Firefox from Mozilla or from Fedora has almost nil speed difference, and Firefox running on Wine is faster than native Firefox. Opera lags behind, but we're inclined to believe that number would increase if a better build was used.

Is this the end of the story? Probably not - we expect some commenters will come up with some other reason why the slowdown is clearly Nvidia/ATI/someone else's fault. Go on, folks: try the benchmark yourself. If you're running Linux, install Wine and try Windows Firefox on Wine and make up your own mind.

You should follow us on Identi.ca or Twitter


Your comments

Opera Wine?

What about trying to run Opera windows with Wine? If that is also faster than Opera for linux, then we have a trend...

PCLinuxOS 2008

I got 178 with pclinuxos 2008 seems it is not linux but Fedora is the culprit. My impression using Firefox in pclinux was always faster than in windows or other linuxes

Score: 178

Richards: 183
Deltablue: 221
Crypto: 153
RayTrace: 141
earlyBoyer: 234
RegExp: 156

some data points

When I run the benchmarks in my FF on ubuntu (3.0.6 FF with plugins and other stuff running on the machine like VM's etc, so not a very clean test)

I get a result of 110.

When I preview the page inside aptana studio I get ~190
when I run the page serverside with jaxer I get ~190 also.

I think the results are skewed by the browser wrapper/chrome, and not indicative of the js engine per se (although still something i'd like looked at)

The tests themselves may be suspect for comparing an interpreted JS against a VM'd one, and could be skewed towards optimized features of the V8 engine like scope chain flattening.

but it clearly shows the linux FF is not well optimized vs the windows one.

PCLinuxOS 2008 Konqueror and Opera

Well Konqueror is fassst but it took much longer before the benchmark run the test???:

Score: 27.5
Richards: 16.6
DeltaBlue: 18.9
Crypto: 12.2
RayTrace: 56.6
EarleyBoyer: 35.8
RegExp: 55.9

Opera:

Score: 146
Richards: 104
DeltaBlue: 122
Crypto: 99.7
RayTrace: 334
EarleyBoyer: 544
RegExp: 42.5

gentoo amd64 wine vs native

gentoo amd64; firefox:3.06; opera: 9.63; wine: 1.1.14

firefox-native: 205
firefox-wine: 223

opera-native: 190
opera-wine: 224

Timer API difference

Does anyone thought about how the accuracy of timer api between wine and native linux can influence results?

Btw as far as I know google's test uses recursion heavily which is not much optimized by tracemonkey.

Oops

Thought lower results are better

What about 64 bit?

What are the results when using a 64 bit OS?? with 64bit Firefox on linux, and Win Firefox 64 bit on WINE?

Also does this changes from Distro to distro?

what about opera lin, and opena win?

there are still too many questions not answered, how does the CPU affect in all this. Intel <> AMD

Will you get different results if you try different video cards?

wow

wow

OMG I'm going to go home and

OMG I'm going to go home and cut myself now. *sob*

Apportioning blame

These numbers are particularly interesting because of what they suggest *isn't* the problem.

If ff+wine is almost as quick as ff+win, doesn't that mean that the linux kernel and X are off the hook here? Doesn't it also hold that gcc is quite capable of compiling linux, x and wine to be quick enough to run ff+wine at near-native speeds, and therefore gcc itself is not broken (as I saw suggested on slashdot), even if perhaps it is not being used optimally to compile ff+linux?

And most importantly, doesn't that also mean that the wine people have done a f**king amazing job if we can expect near native performance with a huge app like firefox in a clean-room reverse-engineered reimplementation of something as stupendously huge as the win32 api?

I have always felt that ff+linux was a bit too sluggish. It's kinda usable, but the full-page zooming feature is ridiculously slow and scrolling is a joke. It's imperative that the source of the problems here be identified and rectified, but it seems like the ff+wine vs ff+linux numbers do vindicate some components of our os of choice, and demand vociferous praise for others.

Chrome

In Firefox I get 250, in Opera I get 262, in Chrome I get:

Score: 2446
Richards: 3182
DeltaBlue: 3729
Crypto: 2625
RayTrace: 4370
EarleyBoyer: 5448
RegExp: 288

wtf?

Linux and X have NOTHING to do with this

Yeah it means the Linux kernel and X have nothing to do with it.

Why people are trolling X because of this article, I will not know. There are a lot of fucking clueless people who just sit and talk shit about MIT X architecture, and never even looked at a line of code in their lives. Just a bunch of fucking morons who hate Linux because they are too stupid to appreciate it.

about compiler optimization

To the poster who said that "There is only so many ways you can write an optimizing compiler". It is exactly the opposite. It is actually a fun fact - it is proven that there are infinite number of ways to write an optimizing compiler.

---

Realistically, there is only so many ways you can statically optimize a piece of code. Ultimately a compiler is basically a translator, it takes like C or C++ code as input and outputs assembly or machine code. The problem is C or C++ is not very descriptive, it doesn't really tell the compiler what people will use this program for or how it will be used. So this limits the kind of optimizations that can be done. GCC does many optimizations already, maybe not ALL optimizations that are possible statically, but it does a hell of a lot of optimizations.

Don't blame Mozilla, distros compile their own packages

Mozilla creates a vanilla linux version of the browser as a tarball with every release. In almost every single case, distros ignore this build and instead grab the source tarball (or directly from hg/CVS) and compile their own packages from scratch.

Yes, Mozilla concentrates more effort on improving the speed of Windows through compilation tweaks, but where's the incentive to do otherwise? Different distros have different (and sometimes contradictory) needs for building and packaging. There is not a one-size-fits-all answer for Linux builds.

firefox

It's clear that this is a problem with Firefox, GG people trying to pass the blame to X or Linux like the previous article.

Anyway It looks to be fixed in Firefox 3.1 choobs

Any moron knows that Firefox Linux is a Pile of Sh*t

And it's been that way forever. And it's gotten worse ever since the pre-1.0 builds.

It has nothing to do with compilers, drivers, blah blah. The native linux code just sucks. Simple.

If you really want to be blown away, try using Chrome for a couple weeks. Then use Linux Firefox. The slowness is hard to miss.

Logical Fallacy collection

[some take pride in saying that the x11 server is as old ...]
Argumentum ad novitatem. something is 'bad' because it's old... (if the argument was really about complexity, why mention old?)

[every sane programmer knows ]
aka "No True Scotsman ..." fallacy

[so much faster because the platform is mature.]
ironically, the author use the contrary of it's first fallacy, in the same sentence. this time it is an "Argumentum ad antiquitatem". it is old(er) - or should I use the euphemism, 'mature' - hence it is better.

Benchmarks on OS X

Firefox (Shiretoko 3.1b3pre w/ tracemonkey): 103

And just for kicks...
Safari 4 (5528.1): 327

Insane difference and from reading the posts, it seems like Firefox is just as slow on OS X.

All this babble...anyone have something technical to say?

Man...this is such an interesting subject, and reading back I see 2 or 3 technical posts and the rest is just emotional crap.

I guess I'm contributing to the noise here because I also did not investigate, I'd just like to say shut up and do some investigating before posting shit.

Now taking my own advice....

Linux is bloated and slow

People should just stick to real world operating systems like OS X and Windows.

K-meleon is ten times a

K-meleon is ten times a better browser than is firefox. I wonder how it would fare in such a comparison?

Congrats! You made it!

If you've read the sodding drivel above, and are reading this line now, congrats.

Let me summarize for you as somebody with a B.S. in Computer Science who isn't a drooling mouth breather:

1. Clearly if the WINE version is faster this excludes X and Linux as possible sources of the problem since WINE also runs under X and Linux.

2. The WINE version is probably ALSO compiled with GCC, so GCC isn't the problem unless the distros are using a whole lot of stupid when they compile Firefox Linux.

3. The person who said that X is bad because it is "old" and Windows is good because it is "mature" in the same argument is a complete idiot who didn't realize they were contradicting themselves.

4. Sounds like the Linux-specific code for the Linux version of Firefox sucks, and the shoddy programmers who are to blame should be flogged. (Or left to fix it, which it sounds like they may have already done for Firefox 3.1)

Hope this helped.

well, one correction

well... obviously windows firefox wasn't compiled with GCC, but WINE and X were.

It's more than likely not the compiler...

Having used all of these compilers, and more importantly *read their documentation*, I can tell you the only reason one would want to use ICC is for the extended instructions.

Things like SSE2, etc. Unless your doing a *lot* of FPU calculations, there will be 0 difference between GCC and ICC. Although, in reality, if you're needing that much performance, most people will hand write the SSE2 code in assembly, as generally you can squeeze out a lot more performance than any C compiler will.

In some circumstances GCC will outperform ICC and MSVC. ICC can only really shine for lots of FPU calcs. MSVC doesn't really shine at all, although for 99% of code there will be no difference between any of them.

Compiler optimisation is not a new science, and while new things are still being developed, no one lets their compiler lag behind the trends.

Cannot be X or Video driver.

Wine also runs on X and the same video card, remember?

It's just that firefox don't care much of code quality on Linux because of the monopoly it enjoyed on the Linux platform. Now more and more Linux machines switch to webkit based browsers, firefox will get some real competition against it.

cool benchmark

thanks for putting me up on that. here's what i got:

Firefox 3.0.6
Score: 81.8
Richards: 79.8
DeltaBlue: 98.2
Crypto: 83.2
RayTrace: 68.8
EarleyBoyer: 113
RegExp: 59.2

Webkit r40955
Score: 930
Richards: 1652
DeltaBlue: 1101
Crypto: 1443
RayTrace: 767
EarleyBoyer: 1767
RegExp: 182

Mac OS 10.5.6 running on a lowly but sexy MacBook Air.

javascript?

Maybe don't do your time-intensive computing tasks with javascript then. I've heard java and C++ are popular.

kubuntu 8.1

kubuntu 8.1 kde-4.2
2.6.27-11
ff 3.0.6 = 107
ff 3.1 = 90.1
ff 3.2 = 75.6
konqueror = 85.1
swiftfox = 109
opera = 88.8
flock2.0b3pre = 103
arora0.4 = 39.4
epiphany-gecko = 80.7
epiphany = 65.1
midbrowser0.3.0rc1 = 104

wine
opera 9.63 = 116
ff3.0.6 = 123

ff3.1&3.2 from fta ppa swiftfox and opera from repo on there sites
after each test with the tested browser open i ran ff3.0.7(left open for all tests) to ensure that no background process was skewing the result each time it came back with 107

Firefox is by default starting in offline mode in ubuntu

It's very annoying,although it's pretty easy to solve the problem manually within 5 min. But why? is this a fault of ubuntu or firefox??

Compiler?

To rule GCC vs MSVC difference out, please recompile the Windows version with GCC and compare with the Linux version compiled with the same version of GCC.

So much for meaningful benchmarks

Just ran the V8 benchmark for FF 3.0.6 and IE 7.0 with the following results:

IE 7.0 31
FF 3.06 115

Flash is even better in wine

After reading this article I tried to install the windows version of firefox on my machine. Surfing the web I also installed the flashplayer. And here came the big surprise. Flash applications started to fly.

The native flash is very slow and uses lots of ressources. Playing games like dogfight2 or watching videos on youtube is painfull with my old processor.

But the windows version of flash works like a charm. The cpu usage halves and the games start to be playable.

So What?

I always knew that windows is superior than linux but linux is very good considered it is a free software. It is not a big difference, why bother?

Firefox 3.0.6 openSuse

Firefox 3.0.6
openSuse 11.1
score - 255

Could it be something internal to wine?

Anyone who runs wine from the terminal would attest that wine still does not implement a lot of calls to windows library routines.

If a program it is running makes one of these missing calls, it usually just displays a line like "fixme: etc, etc..." to the screen.

My point being, if an application like ff makes enough of these calls in the background, wouldn't wine be saving it time by not calling these library routines?

Another benchmark

I'm feeling pretty sad with this.

Here are my own benchmarks using Google's V8 test:

Firefox 3.06 for Windows on Wine, from Mozilla

Score: 216
Richards: 198
DeltaBlue: 270
Crypto: 153
RayTrace: 224
EarleyBoyer: 290
RegExp: 189

Firefox 3.06 for Linux, from Mozilla

Score: 174
Richards: 195
DeltaBlue: 216
Crypto: 164
RayTrace: 146
EarleyBoyer: 218
RegExp: 125

Firefox 3.06 for Gentoo, built with -march=prescott -O2 -pipe -Wall -j3

Score: 182
Richards: 177
DeltaBlue: 210
Crypto: 173
RayTrace: 144
EarleyBoyer: 239
RegExp: 166

I've run them several times and the values change by no more than 5 points. I'm inclined to support the opinion that such a difference is due to the underlying code-base for Linux. Notice how derisory is the difference between my Firefox build, with my computer's settings, and the build by Mozilla (about 10 points), in comparison to running Firefox on Wine…

Of course, this is not a scientific approach, and I'd like to see you guys doing it. :-)

On my ubu 8.10 firefox is

On my ubu 8.10 firefox is very slow. I've got nvidia 8600gt, intel t7500 and 4gb of ram. In my school we've got kubuntu with 512 mb of ram, and some integrated graphic card and firefox is very fast. WTF?!

Why do we care about the V8 benchmark?

Seriously, is optimising for JavaScript really the best use of our time?

How do we EXPERIENCE the slowness?

Personally, it drives me nuts that I have to wait for the hard disk to churn when starting up the app or opening tabs, or anything else. The whole damn thing is too fat (GTK is an ugly, heavy library, viva Qt!) and not asynchronous enough.

As a side note, I think a lot of optimisation in many apps has to do with disk activity. A use case: right click on a desktop item and you might get a popup list of actions. Some of these actions are known in advance (delete, run, etc) and some are dependent on external applications (encrypt with GPG, etc).

I have to wait for all the options to be discovered before the popup list is shown. Why oh why can't items be added asynchronously? It would be more responsive, which is IMHO more important than having the complete list appearing intact. (I'm the kind of guy who never enables smooth scrolling etc.)

So what's my point? My point is that in this example optimising the GUI toolkit to show popups faster won't do squat if we're waiting for the file type to be inspected before we issue the display call. The solution is to display an incomplete list and update it irregularly.

There must be countless bits of slowdown in firefox and other applications that have sweet sod all to do with compiler flags and kernel settings.

Why optimise JavaScript? As long as it's faster than the internet connection it's fast enough. The important thing is to BE FASTER THAN THE USER.

I do, however, think it's tragic that gcc isn't a great optimiser and/or Firefox has lazy porting.

I just installed Firefox for

I just installed Firefox for Windows and the Flash plugin under wine on UBuntu 8.10. The performance of both, Flash and Firefox, were way better than a native Firefox on Linux.

This is crazy! Mozilla developers really should do something about that.

Anothet test results

Gentoo Linux Amd64
gcc 4.3.3, FF-3.0.6, wine 1.1.16
CFLAGS="-g -DNDEBUG -march=k8 -O2 -msse3 -pipe"

Google V8 Suite (draw)
Firefox Wine (1.1.16) 150, 146, 150
Firefox Linux 150, 150, 145

SunSpider (Linux Native wins)
Firefox Wine (1.1.16) 4731.8ms +/- 0.7%
Firefox Linux 4403.0ms +/- 1.2%

My results

Ubuntu Firefox: 99
Wine Firefox: 160
Epiphany: 167
Wine IE 5.5: Could complete 38% only

The beginning of the end

This is just one more example of the unclothing of Emperor Linux. Can anyone suggest for a moment that the Firefox team wouldn't use every possible tool or technique available to ensure that the Linux distro performs as well as possible? Of course not. The issue clearly lies with the quality of the operating system. But of course the infected carriers of MDS (Microsoft Derangement Syndrome) still see glittering royalty in a code set that looks like a Rube Goldberg contraption fastened together with thousands of frayed strips of duct tape.

Mozilla hates Linux. That

Mozilla hates Linux.

That explains why Firefox is so slow on Linux.

Very interesting results.

Very interesting results. I've been using Ubuntu for a few years now as my main OS and I've found the main weak point of the later builds is Firefox. It tends to crash (i.e. freezes and needs force quit to close it) or just be really slow at doing some things. My (entirely empirical) observation was that the threading doesn't seem to work as well as it should, so a 'busy' tab can stop things working in other tabs. Also very poor when in contention for bandwidth with, say, bittorrent, so that pages that should load, albeit slowly, never load at all. By contrast when I boot XP to play games and end up doing a bit of browsing things are far snappier, looks like you may be homing in on one possible cause.

SPM

If Firefox on WINE on Linux runs at nearly the same speeds as Firefox on Windows, then that is definitive proof that Firefox's slowness on Linux is nothing to do with Linux or X rendering, but due to the Linux version of Firefox alone. I also doubt if it is anything to do with the compiler used, since the effect sould not be that dramatic. I think it is due to Firefox not being optimised on its Linux version.

and factor responsible for slowness is..

GTK

That's right - it's not secret that this joke-toolkit written in C many years ago to provide free and open source (as of then) competition for Qt is ironically much less responsive and laggy than Qt written in C++ (especially 4.5 with raster renderer - and OpenGL renderer is WIP).

Fortunately, in March 2009, Qt 4.5 will be released and licensed LGPL - that being said - the only reason GTK ever existed will cease to be and GTK should die along with it.

Stop blaming compiler and compilation parameters - I'm using Gentoo for many years and I know well enough that enabling mindless optimizations is not a factor for such dramatic speed improvements - especially comparing to default -O2 which already enables most sane optimizations and is far good enough.

Graphics

I don't know how much graphics the tests do, but an interesting fact of X11 is that it is very slow (each rendering operation is to be done in a separate process).

Qt 4.5 toolkit offers an option for pure software rendering (i.e. in the process of the application), and contrary to what one would think (X11 having hardware acceleration etc.), that software rendering is actually many times faster.

Now if GTK+ toolkit had such a software based rendering mode...

Javascript optimisation is a work in progress.

With benchmarks relating to javascript execution speed (using JIT compilation), one should bear in mind that the lightening increases in speed that have occurred very recently are a work in progress, and the full optimizations have only been introduced to Windows so far - Google Chrome is Windows only so far, as is I believe the Firefox optimizations.

Hence is is a bit of FUD to claim as some have done that this proves Linux is slow compared to Windows - wait a few months for the javascript optimizations to the Linux versions and Linux benchmarks will match or beat Windows.

For comparison look at how IE6 on Windows (not optimised for javascript) runs on Windows compared to Firefox on Windows

Quote/

Just ran the V8 benchmark for FF 3.0.6 and IE 7.0 with the following results:

IE 7.0 31
FF 3.06 115

/Unquote

On the same basis, one could run Firefox 3.06 with the javascript on Linux under WINE and compare it to IE6 on Windows and claim that Linux is more than 3 times as fast as Windows - fodder for the fudsters, shills and fanboys, but disproved by the fact that Firefox for Windows running under WINE under Linux is as fast as Firefox running natively under Windows.

filesystem ext3?

I remember there being an issue with firefox and ext3. I wonder if there's an issue with the caching behavior of the browsers?

I suppose testing to see if the results are different on ReiserFS or ext4 and seeing if the benchmarks change when the caching is disabled...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar