Compiler, Linker, Assembler and Loader: an overview

It is nice for a programmer press that shiny green play button on top of Visual Studio and see his/her application run in the current machine. But, what kind of “magic” is hidden underneath that button? This blog post is a gentle overview of the process that is behind the compilation/running task based on my curiosity on the topic; I will focus on C++ and g++ compiler-driver.

The Four Horsemen

Our PC doesn’t understand C++ or any other language, except for machine code. The machine code is a sequence of instruction written in binary that can be executed directly from the CPU. So, how is our C++ application able to run if it’s not written in machine code? If we are able to run our application is thanks to four main tools: compiler, assembler, linker and loader. A brief overview of the building process is depicted in the following figure.

 

CompilerLinkerandAssembler

 

Compiler

Before our source files are given to the compiler, a preprocessing stage occurs. In this stage the preprocessor deals with include-files, conditional compilation instructions and macros and produces a file that is then given to the compiler. Header files are not seen by the compiler; the preprocessor, in fact, replaces every #include statement in .cpp files with the code written in the respective header file. Let’s make a small test.

Test.h

#ifndef TEST_H
#define TEST_H
// A test class, containing two functions and an integer member.
class Test{
public:
    Test();
    Test(int value);
    int getMember()const;
    void setMember(int value);

private:
    int member;
};

#endif

Test.cpp

#include "Test.h"
//Test class implementation
Test::Test()
    :member(0)
{}

Test::Test(int value)
    :member(value)
{}

void Test::setMember(int value)
{
    member = value;
}

int Test::getMember() const
{
    return member;
}

main.cpp

#include <iostream>
#include "Test.h"

int main()
{
    Test test;
    std::cout << "The int member is currently: " << test.getMember() <
    test.setMember(10);
    std::cout << "The int member is currently: " << test.getMember() <
   return 0;
}

Now let’s open the command line and use g++ to see what is the preprocessor output (you can use whatever C++ compiler you like).
The command to use is

g++ -E Test.cpp -o preprocessed_test.ii

which will only preprocess (-E) our Test.cpp file and put the result into preprocessed_test.ii (-o flag). The extension .ii is not casual: according to the gnu documentation, ii files are (C++) files that do not need to be preprocessed.

If you open preprocessed_test.ii, it will look like this.

Screen Shot 2015-04-01 at 17.20.32

If we want to preserve our comments, just add the -C flag to the previous command. Let’s do the same for our main file, with

g++ -E main.cpp -o preprocessed_main.ii

Examining proprocessed_main.ii we can see a very long file, which is the result of including iostream header file.

The compiler is fed with the preprocessed file we just generated and, in turn, it creates assembly code. In our experiment, we can use

g++ -S preprocessed_test.ii -o assembly_test.s

g++ -S preprocessed_main.ii -o assembly_main.s

to generate the assembly code. The extension .s denotes assembler code. Assembly_test.s will look like this

Screen Shot 2015-04-01 at 17.41.03

This is the end of the proper “compiling” stage, as we reached assembly code.

Assembler

The assembler aim is to build an object file. An object file is basically our source file translated in binary format. To assemble our test, run

g++ -c assembly_test.s -o object_test.o

g++ -c assembly_main.s -o object_main.o

With the -c flag, we are telling g++ to just compile (in this case to just assemble) our input, skipping the linking step, which will be discussed in the next section. An object file can come in different forms, such as ELF (Executable and Linking Format) on Linux systems and COFF (Common Object File Format) on Windows. An object file is made of sections, containing executable code, data, dynamic linking information, symbol tables, relocation information and much more. If we want to inspect what an object file is and what it contains, we can use tools such as objdump or readelf. On my macbook I have installed the macports version of objdump, gobjdump, used for our example.

gobjdump –all-headers object_test.o

Will give us human-readable information about the object files:

Screen Shot 2015-04-01 at 17.55.51

What are all those sections and what is their meaning? Some of them are described below.

  • .text section contains the executable code; this section is the one affected by compiler optimizations;
  • .bbs section stores un-initialized global and static variables;
  • .data part is responsible to store initialized global and static variables;
  • .rdata contains constant and string literal;
  • .reloc holds the required relocation information of the file;
  • The symbol table contains pairs of type <symbol,address>. It is used to lookup the addresses of a symbol in the program;
  • The relocation records helps the linker to adjust other sections.

Let’s now look at a simple assembly program (x86) here

Do you see something similar?

Linking

The linker produces the final executable, that can be run by our machine. As the word suggests, the linker links all the object files together, allowing separate compilation. In short, the linker calculates the final address of the code and resolves all the external references present in each object file.

 

linker

In our example, we have two files, main.cpp and test.cpp; in our main function there are some references to functions defined in test.cpp, so we need to connect those two files somehow. The linker takes as input our object files and other compiled source code (such as libraries) and gives us an executable as output, resolving all the references between the files.

In order to complete its job, the linker uses the relocation records and the symbol table: relocation records are used to find the addresses of the symbol referenced in an object file, for example if we call the Test setMember function in our main, we will have a relocation record for it in main.o so that the linker can substitute the actual instructions from test.o; the symbol table is used to resolve all the references among the different object files. Let’s see how this works briefly.

Executing objdumb with object_main.o as input and search for the Symbol Table (or, alternatively use the –syms flag), we can see that some symbols are undefined.

Screen Shot 2015-04-01 at 19.46.29

These symbols refer to functions defined in Test.cpp. In fact, if we examine object_test.o, the same symbols will be defined, take the setMember function as an example.

Screen Shot 2015-04-01 at 19.44.59

To generate our executable run:

g++ object_main.o object_test.o -o program

If we now examine program with objdump, we will have all the symbols defined. Thanks, linker!

Relocation

The main effects of the linker relocation is to produce the final addresses for the labels present in our program. This means that the addresses assigned before the linking stage are temporary and relative. Relocation can be split in two sub-tasks: section merging and section placement. The first is simply a matter of merge the same sections present in our object file into the executable. This means that in our file, the .text and the .data sections will be the result of the merging of the same sections in the object files; the addresses will be affected by the merging. The section placement is to set the starting memory address where all the addresses have to refer to: usually they are relative to the address 0, but after the placement stage all the addresses are shifted by the starting index.
The following image, taken from this page, will summarize and visualize in a great way the relocation passes.

relocation

Linker and Libraries

In our code, it is possible to include libraries and pre-compiled code; this means that the linker should consider the library as well to produce our executable. We can define a library as a collection of object files, that can be used in external programs. With static linking, all the files are linked during link time, resulting in an executable file. All the symbols and the information needed by the program is known before running our code; this type of libraries have usually the .a extension. Since we need all the information stored before run time, static linking produces big files.

Another option is to use dynamic linking. In this case, the linker places information into the executable to inform the loader where to find the library, which is bound in runtime, just before the program is executed; this type of libraries have usually the .so extension. So, why one should use dynamic linking? Dynamic linking makes the size of the executable smaller; a library can be updated without re-linking; it permits to share read-only library modules, so that less memory required.

Loader

After we came up with an executable file, we can run it on our machine. The first step is to load the program in memory (RAM) and this task is accomplished by the loader. In the first place, the loader validate the program by computing the memory requirements and checks its data type, instruction set etc; secondly it allocates the memory needed, copies the .text and the .data sections into the RAM, fills the program stack with program arguments and initializes registers, setting the stack pointer on top of the program stack and cleaning all the rest. The final operation is to execute the main function.

The program is now running on our machine.

References:

https://gcc.gnu.org/onlinedocs/gcc-4.0.4/gcc/Overall-Options.html

http://www.bravegnu.org/gnu-eprog/linker.html

http://www.tenouk.com/ModuleW.html

Game Engine Architecture by Jason Gregory