Compile source code - and solve problems

LXF

Building software from source - that's a bit old-school, isn't it? Who wants to wrestle with the command line, hunting down dependencies and coaxing the GCC compiler into running properly? Well, it does sound like a strange thing to do in this world of binary packages and online repositories.

We have thousands of packages available via the internet, all neatly compiled for our distros, thereby usually nullifying the need to get down and dirty with a Makefile. Or so it seems... Read on to find out why you may want to compile a program from its source code, and deal with the problems that can crop up.

As great as they are, binary packages have a lot of limitations that can only be overcome by compiling a program from its source code. Here are some of the benefits of building by hand:

  • Timeliness Get software as soon as it's available. When CoolApp 2.1 is released, your distro will probably have only version 2.0, and (unless it's a rolling-upgrade distro like Gentoo) you'll have to wait until the next major release of your distro to get the new app. But by building from source, we can get the newest version of anything when it's released.
  • Features In the interests of stability, distro vendors often disable experimental features when making a package, leaving you with a rather plain (albeit reliable) piece of software. When we go down the source-code route, however, we can turn on extra goodies - it's our choice!
  • Optimisations When you compile from source, you get the chance to boost the program for your machine. Binary packages are built for a wide range of x86 processor types. Using certain compilation options, you can build apps specifically for your exact model of CPU, helping to wring extra performance from your PC.

To prove that it's not quite as difficult as it sounds, let's see how this works. The skills you learn here will put you in good stead for your Linux (and Unix-using) career, as the vast majority of programs can be compiled in this manner.

We're going to take a stock, unmodified Ubuntu 8.04 installation and install the Audacity sound editor, enabling some extra features along the way. Audacity is a good choice because it usually throws up some strange errors during the compilation process, so you'll learn how to deal with every eventuality.

The stock Ubuntu Audacity package: good, but we can add a couple of effects by compiling from source.

The stock Ubuntu Audacity package: good, but we can add a couple of effects by compiling from source.

Get prepared

First off, we need to get hold of the Audacity source code. You can find this on the project's website at http://audacity.sf.net. Save the tar.gz file of the latest version to your desktop (eg audacity-src-1.2.6.tar.gz) on to your desktop, then open a command line window (Applications > Accessories > Terminal) and enter the following:

cd Desktop
tar xfvz audacity-src-1.2.6.tar.gz
cd audacity-src-1.2.6

The first command switches us into the Desktop folder, and the second command extracts the compressed .tar.gz archive. (Note that for archives with a .tar.bz2 suffix, you should use tar xfvj instead.) Finally, we switch into the newly created source code directory.

Enter ls to see the list of files in the expanded archive. Along with various scripts (highlighted in green) and sub-directories (blue), you'll see a README.txt file. It's always worth having a quick peek at any file called README.txt or INSTALL.txt, as it may have useful information on building the program.

Enter less README.txt to view the file, and hit the Q key to quit. In this case, we don't need to read through it all - we can get straight to work on the compilation process.

The configure script will prepare the Audacity source code for compilation. Enter this command:

./configure --help

This executes the configure script (the ./ is required, as we're running it in the current directory), telling it to spew out all available compilation options. If you scroll up through the terminal, you can see that there's a huge range of options available to us. But don't fret - we can happily accept the defaults.

Here's the output we see when running ./configure --help - you'll see all the pre-building configuration options that are available.

Here's the output we see when running ./configure --help - you'll see all the pre-building configuration options that are available.

We're going to make a couple of small changes though: firstly we're going to enable the SoundTouch library, which provides extra pitch and tempo manipulation tools that aren't included in the default Ubuntu package.

Another tweak we can make is to set the location of the installed program when it has been built. To do this, we use the --prefix option with the configure script. For instance, to install Audacity in the /usr/local directory (away from other software and easier to maintain), we use --prefix=/usr/local.

As mentioned earlier, another great benefit of compiling software from source is the choice of optimisations. By setting the CFLAGS (for C code) and CXXFLAGS (for C++) environment variables before we run the configure script, we can select the exact GCC compiler optimisations we want to use. This is a complicated subject, but we can get an extra speed boost very easily. Consider these commands:

export CFLAGS="-O3 -march=core2"
export CXXFLAGS="-O3 -march=core2"

Here we're setting up the aforementioned environment variables with optimisations suitable for the Intel Core 2 CPU architecture. The -O3 bit at the start has a letter O, not a zero, and requests the third level of optimisation. (Most distributions compile their packages with -O2 for slightly smaller executables and easier debugging.)

If you have a machine with a AMD64 chip, use -march=athlon64, and if you're not sure, just use -march=i686 for wide compatibility. You can consult the GCC manual page (man gcc) for a complete list of supported CPU architectures, but it's heavy reading!

So, we have our optimisations set up, and we're ready to run the configure script. Here we go:

./configure --prefix=/usr/local --with-soundtouch=system

This tells the script that we want to install into the /usr/local directory, and enable the SoundTouch option that was listed when we ran ./configure --help before. (The 'system' part for the soundtouch option indicates that we want to use the version of SoundTouch available in the distro, as the one supplied with Audacity is broken.)

Why compile at all?

If you haven't even glimpsed at a chunk of source code before, you may be perplexed by some of the terminology in use here, so let's clarify the terms and processes.

Compiling is the process of converting human-readable source code - a software recipe that could be written in any high-level programming language - into the binary instructions used by a specific computer's processor.

For the most popular programming languages, C and C++, the GNU Compiler Collection (GCC) does the hard work of converting the source code into a binary executable file. It reads the source files, translating the C or C++ code listing into machine code instructions for your CPU.

Different CPU families have different types of instructions; for instance, if you compile an app's source code to run on an Intel processor, the resulting program won't run on a PowerPC. You need to compile it for each type of processor on which it will be used.

Note that for interpreted languages, such as Python, Perl and Ruby, each line of the source code is translated on the fly. With those languages, there is no compilation process - the language interpreter steps through the program line by line.

Grab dependencies

Oops - something's not quite right. After you've run the ./configure command, you'll get this error back: 'C compiler cannot create executables'. We need to install a development toolchain to build binaries from source, so enter: sudo apt-get install build-essential

On Ubuntu, this installs a meta-package that pulls in all the tools we need to compile bare-bones command-line programs. If you're running another distro, look for GCC, G++ and Binutils in your package manager, and go ahead and install them. Now run the previous configure script again. (A handy time-saver for those new to the command line: hit the up cursor key and you'll be able to retrieve previously entered commands.)

Now the script will proceed more healthily, but it will stop and inform us when it can't find a program on the system called wx-config. Audacity uses a graphical toolkit called WXWidgets to render its interface, and we don't have it installed, hence the error message that you'll see. So we need to grab WXWidgets from the Synaptic package manager.

Before you run off with the mouse, though, it's important to note that many Linux programs, especially libraries and toolkits on which other programs depend, are supplied in two versions.

One version is the run-time library that a program can link to; the other is a bundle of development libraries and header files used for compiling software. So, open Synaptic (under the System > Administration menu) and search for WXWidgets.

Scroll down the list of results and you'll see libwxgtk2.6-0. This is the run-time library on which WXWidgets-using programs depend - it provides a GTK layer on top of the WXWidgets framework. Check that for installation, and also check libwxgtk2.6-dev, which will provide the necessary development bits and bobs for our source code build, and then apply the changes. (In other distros, these packages will have similar names, so search for WXWidgets or wxgtk to find them. Development packages will almost always end with a -dev or -devel extension.)

We can install an app's dependencies with a couple of mouse clicks in Synaptic.

We can install an app's dependencies with a couple of mouse clicks in Synaptic.

Now re-run the configure script. You'll see that Audacity requires an older version of WXWidgets - namely, 2.4.x! This is pretty much the standard process for building from source code, which is why we've illustrated it for you here; it often takes a bit of trial and error to work out exactly what library dependencies and versions are required - and it can sometimes be frustrating. So, head back into Synaptic, remove WXWidgets 2.6, and install version 2.4 (don't forget the -dev package).

The magic of checkinstall

If you've got several PCs, or you're an administrator in charge of a large number of corporate machines, you probably won't have time to follow the whole compilation procedure for every single computer. Thankfully then, there's an RSI-eliminating tool that enables you to create binary packages of your source-built programs.

Checkinstall, available from http://tinyurl.com/2uodas generates Deb, RPM and .tgz packages for Slackware.

After you've installed Checkinstall, run the ./configure script for the software you want to build as described in the main guide. Then run make to compile the software, but don't run the usual make install command afterwards. Instead, as root, run checkinstall.

You'll then be asked a variety of questions, such as the type of package you want to build, and whether you want to include any documentation or not. When Checkinstall has finished its work, you'll have a binary package that you can then install on as many other machines as you like (providing they're running the same distro and have the same libraries installed).

Build the code

Run the configure script once more and you'll see that everything goes smoothly this time. For other programs, you may have more dependencies to sort out before you can build - as mentioned, it's all a trial-and-error process.

For instance, if a program's configure script complains that it can't find the PNG development libraries, look for a package named something like libpng-dev. If you're building a Qt-based app, you'll need libqt-dev. Often the README file will provide a full list.

The command we need to kick off the GCC compiler is (drum roll please)...

make

Simple as that. The 'make' utility consults a newly created file called Makefile, which details the order in which the program's source code files are to be compiled.

Depending on the size of your program - and speed of your machine - the compilation process can take anywhere between 10 seconds (for a small command line app) to several hours (for a leviathan like OpenOffice.org).

During the compilation phase, you'll see that GCC is being executed using the CFLAGS and CXXFLAGS settings we defined before. You may also see some warning messages about non-strict code, but those are nothing to worry about.

You will, however, get a bit of a shock when the compilation process halts abruptly with an error. You'll see that SoundTouchEffect.o couldn't be compiled - does that mean SoundTouch is missing?

Well, yes, and the configure script should have alerted us to that. But not all is perfect in this world, so we have to fix it ourselves. You may recall, back at the start, that we passed the --with-soundtouch=system option to the configure script, instead of =local, because the version supplied with Audacity is broken. (You would've found out that =local is broken, so we've saved you a bit of time!)

So, configure let us down a bit there. But no big deal: open Synaptic, search for libsoundtouch and install it (along with its -dev sibling). We don't need to re-run the configure step again, as that succeeded, so enter make again and watch the output.

Now you'll see that it can't find FLAC. D'oh! So fire up Synaptic and install the libflac-dev library package. Enter 'make' once more, and you'll see that the compilation progresses a bit further, but then halts with a rather terrifying sequence of errors, including this:

undefined reference to 'soundtouch::SoundTouch::SoundTouch()'

Well, it looks like the compiler still can't find SoundTouch, even though we've installed it and it didn't complain during the configure phase. So during the last part of the compilation, when all of the program's files are being linked together into the executable, GCC doesn't know where SoundTouch is.

We need to alter the Makefile which controls the compilation process. Enter:

gedit src/Makefile

Scroll down to line 31, where you'll see 'LIBS' - that is, the libraries with which Audacity needs to be linked. Add -libSoundTouch to the end of the line (it's case-sensitive), save the file and exit the editor. Run make again. You'll see that the main Audacity binary compiles correctly, but the process hits another wall: it can't find the msgfmt utility to handle documentation.

This time we need to rely on Google: search for 'msgfmt' and 'Ubuntu' (both terms in the same search, but not in quotes as it's not a phrase) and you'll see that other users have had similar problems when compiling programs. You'll discover that it's in the gettext package - but, tediously, the Ubuntu team has both gettext and a gettext-base packages.

The latter was installed as part of the build-essential bundle, whereas the former includes our essential msgfmt tool. Maybe one day distro developers will stop fragmenting programs into a squillion ambiguously named packages, but at least the solution was a quick Google away here.

So, install gettext to get msgformat, and run make again. In the words of Borat, great success! You'll land back at the command line prompt with no errors. All we need to do now is install the newly built Audacity into the filesystem, so enter:

sudo make install

This action needs to be performed as the root (admin) user, hence sudo. On non-Ubuntu distros, you may need to switch to root with su and then enter make install. And now you can run the program:

/usr/local/bin/audacity

If you want to uninstall the program, do sudo make uninstall. You can also remove the source code directory whenever you want - the final program is now in /usr/local/.

Font of knowledge

You'll notice that this version of Audacity isn't looking so pretty on the font front. That's because the version of WXWidgets included with Ubuntu we used built against GTK 1, which doesn't have anti-aliased fonts. Recompiling WXWidgets to use GTK 2 is another exercise that you can try when you've finished this guide!

Anyway, load up a sound sample and click on the Effect menu. You'll see two new options that weren't available in the stock Ubuntu version: Change Pitch and Change Tempo.

Thanks to our geektastic compiling powers, we've got one up on Ubuntu. Eat it, Shuttleworth! Given that it's quite fiddly to add SoundTouch support and get those two extra features, it's not surprising that the Ubuntu team has steered clear.

Et voila: our recompiled Audacity. It may be using Gtk 1, but we now have two funky new effects to play with.

Et voila: our recompiled Audacity. It may be using Gtk 1, but we now have two funky new effects to play with.

Most of the programs you compile will go completely smoothly or just require a few obvious dependencies, but we've looked at lots of issues that can crop up, so now you're trained and ready for any eventuality. Now go forth and compile!

First published in Linux Format

First published in Linux Format magazine

You should follow us on Identi.ca or Twitter


Your comments

Add a reason to compile

Of course being a member of an open source project you are compiling that project when ever there is something new in the svn/git repository to test if the change breaks something else on your setup and reporting back if it does.
Enjoy the Choice

make it easier

there should be a program that compiles code for you.
i've been using linux for over a year and i still cant get most code to compile.

debuggin

dont forget that most binary packages strip any debugging symbols so if you run into a application core or segfault its hard to follow where in the code it died.

IT'S CALLED GCC

man gcc

re: make it easier

there is a program that compiles code for you, its called gcc. without gcc it would be wickedly difficult to translate c or any other language into machine code. i don't read binary very well, and i certainly don't speak it. same goes for hex.

re: make it easier

>there should be a program that compiles code for you.

You should look into Gentoo. The distro is built around the idea of compiling packages from source. Its package manager, portage, automates the configuration and dependency resolution.

If you decide to try Gentoo, keep your old distro installed until your sure you like gentoo. It takes more effort than something like Ubuntu, you have to do more of the configuration, but it can be very enjoyable to use.

uh .. compiling won't necessarily optimize your app

You're probably not running a kernel compiled against your hardware, so arguably the binary package you could just download would be better optimized for the generic kernel provided from your distro handlers ..

Re: This is not for newbies!

Arlnold: Indeed, we wouldn't recommend that newbies compile lots of stuff from source, and in this tutorial we deliberately chose a program that presented a variety of challenges during the build process. After reading this, hopefully intermediate users will be better prepared to deal with any problems that may arise when compiling from source.

GENTOO

seriously, its called gentoo... use it already. Why bother with ubuntu and then compile source? Gentoo is built on this idea from kernel to apps. Gentoo is by far the greatest distro out there if not for these reasons alone.

Yeah but

Not another one of the "Compiling is so easy!" articles

I do have to give credit to this one, though, it does a much better job at showing you the real picture of compiling that the others that just tell you to ./configure and install.

Everything is fine if all the dependencies are easily retrievable from a package manager, but what if they aren't? What if the version you need isn't in the package management repositories? What if you can get a later copy, but it somehow conflicts with something else?

You then have to go manually handle stuff and the amount of time you have to spend on a project grows rapidly. Best case scenarios it really is just ./configure && make && make install, but I rarely run into that.

One or more dependency sinks and the average user has to spend more time installing their little app than they would like to. This is why package distributions rule the game right now.

Re: GENTOO

I can understand why someone might want to compile an application or two, but really who would want to compile everything that they need or want?

Perhaps I am just lazy because I would rather install the program and get to using it.

another reason

Too many dependancies in distribution packages sets means that you have lots of software that you don't need at all.

I have lost count of the

I have lost count of the number of times i have walked total newbies to compiling our project vegastrike lots of dependencies and multiple targets to deal with but all managed to get a running game one even wrote a how to for Ubuntu users on our wiki for the copy/paste gang a day after doing it himself.
Enjoy the Choice

Re: GENTOO

You mean besides being slim and faster than anything else? The only bloat thats there is something you intentionally installed. Everything optimized for your system according to your wants/needs/desires ... who would want to run anything else? Barring those who barely survived toilet training by drowning headfirst that is.

GaryM

Slackware has never failed to compile an app for me. I'd put it right up there as one of the best distros to do this with.

If you are going to compile your apps to enable features why use Ubuntu at all? Keep your system lean and compile just what you need.

Your system will run faster as any Slackware, Gentoo or Arch user will testify to.

This is why I can never feel

This is why I can never feel comfortable telling people that ubuntu (or any other linux distro) is an alternative to windows/osx.

Gentoo: +1

Yup, I use Gentoo also. Besides being infinitely configurable, it's fast and manages all of a package's dependencies. The three benefits mentioned in this article are also realized by using Gentoo: timeliness, features, and optimization.

Perfect to learn off of

Years ago when I first switched to Linux I was using an outdated version of SuSE which being outdated, wouldn't upgrade. Without the ability to get another distro I was forced to compile by hand everything if I wanted to use KDE3, which was just released. After several weeks of banging my head against a brick wall figuring out how to compile programs in GNU/Linux by hand well enough to install damn near anything, I must admit that I got to know the OS pretty damn well. It seems scary (and it is sometimes), but if you _really_ want to know how Linux works, this is how.

...I think it goes without saying that I'm a Gentoo nut nowadays. :-D

...omg if that doesnt put noobs off nuthin will.

...and once you got it compiled, you're ready to be a brain surgen or fighter pilot (two jobs which are a lot easier to do than compiling in linux).

the linux community trods along with their man pages from hell and decades old methods of doing things. ..and thats where it will stay, one foot in hell, fowever.

Of course Linux is fine if you a first class honour student in IT, majoring in psychic learning.

linux = lol

btw Windows is hells doorman.

Excellent article, but it shows the poor state of tools

An excellent article, clearly highlighting the possible benefits and the pitfalls of compiling an OSS application from scratch.

If anything, the article also highlights the problems with Linux and the GCC toolchain for beginners. The point is: the stuff is poorly documented (if at all) and error messages are of the "Clear only if previously known" school of thought. I mean: a message like "C compiler cannot create executables" seems calculated to throw people off. A message like: "C compiler not found (possibly not present)" would be helpful, but unfortunately such details receive no thought. The easiest way is to use a "Howto" that tells you *what* to do and how to check that you did it correctly (but which won't tell you *why* you're doing what you do).

And this Unix setup in which packages are really meant to be installed by system administrators (all dependencies and paths point to places like /usr/local) is decidedly end-user hostile. As is the need to manually add dependencies to the makefile because someone unfamiliar with the (outdated and arcane) makefile syntax is more likely than to mess it up by converting one or more tabs to spaces. (If I got a penny for every user who saved their makefile from an editor that mangles tabs ...).

Besides which, end-users should be able to build and install everything they do in the one place their account has full access: their home directory. So they won't mess up their box and don't have to go around playing sysadmin.

And yes, I consider the GCC man pages the epitome of poor and off-putting documentation. And yes, there is a distro that's focused on making the build-from-source route easier on beginners, but most other distros I know of (Red Hat, SuSE) don't. And SuSE for example uses a subtly different directory structure from Red Hat and others (the location of various startup scripts).

The message is clear: end-users should stick with what their distro has prepared for them; power-users and those who can do simple system administrator chores can venture further.

Re: This is why I can never feel

@Anonymous Penguin, don't be a twit.

Nobody said you had to compile anything. This is an optional extra.

If you want to compile something in Windows you will likely have a much more difficult time of it. First you will have to find a compiler, and you can't just start up your package manager and install it from there, because there is no package manager! You have to find one and download it.

If the software you're trying to compile depends on any libraries, again you can't just install them from your package manager. You have to find the download site for the library and manually install it.

So, although this looks difficult, it's not something that the vast majority of Windows users will ever have to do (or be able to do) on Windows, and it is also not something that the vast majority of Linux users will ever have to do.

Slackware Rules

There is a saying that goes something like this:
If you want to learn Red Hat use Red Hat,
If you want to learn Gentoo then use Gentoo,
If you want to learn Debian use Debian,
If you want to learn Linux use Slackware.

Prove Yourself

Allow me to expand on the "Slackware Rules" post:

When you want to PROVE to yourself that you know what you're doing - build your systems using Linux From Scratch (LFS). If you're a real glutton for punishment, try CCLFS, the cross-compiled version. Once you've built a few, you'll find the control they allow you over your software is addictive.

RE: Slackware Rules

...but I think the important part there is "If you want to learn".
It makes me sad when I hear recycled nonsense like that old "free if your time is worthless". People really have no interest in learning. The computer is just another TV for most.

extra speed boost?

I was going to take some tips from article to compile xmms for my netbook. but after getting to the words "extra speed boost" I realized I just couldn't be bothered.
Compiling from source code mainly doesn't work. it really should be confined to the "last century" part of the operating system story. (mainstream)
Even when I absolutely should compile something thats not working, I'd normally just look for the nearest alternative.

Nice try with the article but some of us (I mean me) just can't be bothered. there are just to many faster less annoying alternatives.

Fail - yet again, another reason I don't compile

checking for FLAC/all.h... no
checking soundtouch/SoundTouch.h usability... no
checking soundtouch/SoundTouch.h presence... no
checking for soundtouch/SoundTouch.h... no
--- Configuring libsndfile
--- Configuring libresample
checking for zip... /usr/bin/zip
checking for wx-config... no
configure: error: "Could not find wx-config: is wxWidgets installed? is wx-config in your path?"
ben@ubuntu-desktop:~/Desktop/audacity-src-1.2.6$ sudo apt-get install wxWidgets
[sudo] password for ben:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package wxWidgets

Quoting myself gets old... :/

"Everyone has different needs of their hardware and software."

Linux isn't mindless "me too"-ism. Stop pretending that what's right for you is right for everyone. Isn't that why you're here in the first place?

Its good and bad

Not for all .................but good for them who needs it.

...

lolwow@linux

Compiling programs from scratch? It's so easy, you only have 57 hoops to jump through! Can you levitate a car with the power of your mind? If so, then this article is for you!

Comparing driving a manual

Comparing driving a manual shift vehicle with using Linux.

At first glance, they seem to be good analogies for each other. But look closer.

Driving a stick shift boils down to knowing which gear to be in at whatever speed you're going. To accomplish this, all one needs is sufficient motor skill to press the clutch, shift to another gear and release the clutch while using the gas pedal with the right foot. A bit hard to do at first, but easily learned given time... like riding a bicycle.

Using Linux requires knowing which libs to get, what options to set, which scripts to grab and execute, learning cryptic lingo for obscure packages/tools/etc. You also need the patience of Ghandi, and you must also understand Einstein's theory of special relativity. Oh yes, you must also be able to bake a potato with the power of your mind.

Put another way, if using Linux were an equivalent analogy to driving shift, you'd be given a variable number of gears to shift into, which changes every few days depending on the barometric pressure and alignment of the stars. Some gears can only be shifted into if you have the right shift knob, which you must purchase, retrofit from an existing shift knob, cobble one together yourself, or beg a complete stranger to mail the parts and blueprints to you - while you're driving the vehicle. The clutch pedal will be smooth one day, and unforgivingly hard the next day, ranging from a push of 10lbs to 100lbs at the blink of an eye. And to switch gears, you must push the clutch in 1/3 of the way, then exactly 1/17th of the way further in for the next gear up - except at high speeds. The definition of high speeds can be found by flipping through the user's manual, which lacks an index. Every 4th page is stuck together with sticky tape, and the latter half of the manual - the user maintenance part - is written in Latin. Downshifting is accomplished the same way, except you must solve an equation on the first law of thermodynamics while juggling baby hedgehogs.

Want Help

Hey guys i want to find solutions of compiler exercises can anyone help me by just telling me any website for solutions? Thanks

checkinstall

Thank you for pointing out checkinstall!

It took 5 hours to get it to compile on my distro, but once I got it working, it has saved me so much time.

Yes, it was a pain to get it to compile. Had to edit c source code. Relied on another site for help. But then I made it work and checkinstall is working wonderfully.

Cheers!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar