Building and Running the Linux Kernel

May 17, 2022 8-minute read

linux • kernel

These are mostly notes for myself.

TL;DR

# fetch the source
git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

# configure the kernel
cd linux
make defconfig ARCH=x86_64
# Enable two options:
# 1. Kernel hacking -> Memory Debugging -> KASAN
# 2. Kernel hacking -> Compile-time checks and compiler options -> Debug information -> Generate DWARF 5 Debug Info
make menuconfig

# build the kernel
make -j $(nproc)

# run the kernel
sudo mkinitramfs -o initfs
qemu-system-x86_64 \
    -kernel arch/x86_64/boot/bzImage \
    -nographic \
    -append "console=ttyS0" \
    -m 1024 \
    -initrd initfs \
    --enable-kvm \
    -cpu host \
    -s -S \
    -fsdev local,path=$(pwd),security_model=none,id=test_dev \
    -device virtio-9p,fsdev=test_dev,mount_tag=test_mount

# debug the kernel (in another shell from the same "linux" directory)
gdb vmlinux -ex "target remote :1234" -ex "c"

Getting the Linux Source

TL;DR

git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

The linux kernel source is found at git.kernel.org, with the main branch being Linus’: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Clone this with --depth 1, unless you want the entire history:

git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Building the Linux Kernel

TL;DR

cd linux
make defconfig ARCH=x86_64

# Enable two options:
# 1. Kernel hacking -> Memory Debugging -> KASAN
# 2. Kernel hacking -> Compile-time checks and compiler options -> Debug information -> Generate DWARF 5 Debug Info
make menuconfig

make -j $(nproc)

You’ve got the code, now to build it.

The first thing to know is that the linux source has a massive amount of options that are controlled by the config file at .config. This file doesn’t exist with a fresh copy of the linux source. A number of options exist to create one from some default settings, a menu config, etc:

$> make help | grep config
  clean           - Remove most generated files but keep the config and
  mrproper        - Remove all generated files + config + various backup files
  config          - Update current config utilising a line-oriented program
  nconfig         - Update current config utilising a ncurses menu based program
  menuconfig      - Update current config utilising a menu based program
  xconfig         - Update current config utilising a Qt based front-end
  gconfig         - Update current config utilising a GTK+ based front-end
  oldconfig       - Update current config utilising a provided .config as base
  localmodconfig  - Update current config disabling modules not loaded
  localyesconfig  - Update current config converting local mods to core
  defconfig       - New config with default from ARCH supplied defconfig
  savedefconfig   - Save current config as ./defconfig (minimal config)
  allnoconfig     - New config where all options are answered with no
  allyesconfig    - New config where all options are accepted with yes
  allmodconfig    - New config selecting modules when possible
  alldefconfig    - New config with all symbols set to default
  randconfig      - New config with random answer to all options
  yes2modconfig   - Change answers from yes to mod if possible
  mod2yesconfig   - Change answers from mod to yes if possible
  mod2noconfig    - Change answers from mod to no if possible
  listnewconfig   - List new options
  helpnewconfig   - List new options and help text
  olddefconfig    - Same as oldconfig but sets new symbols to their
  tinyconfig      - Configure the tiniest possible kernel
  testconfig      - Run Kconfig unit tests (requires python3 and pytest)
  kselftest-merge   - Merge all the config dependencies of
                      kselftest to existing .config.
  configuration. This is e.g. useful to build with nit-picking config.
  kvm_guest.config      - Enable Kconfig items for running this kernel as a KVM guest
  xen.config            - Enable Kconfig items for running this kernel as a Xen guest
  i386_defconfig              - Build for i386
  x86_64_defconfig            - Build for x86_64
  make O=dir [targets] Locate all output files in "dir", including .config

Or you could copy the config that your current linux system was built with, and tweak it from there:

cp /boot/config-$(uname -r) .config
# edit the config you copied from your current linux system
make menuconfig

Or you could use a default configuration for a specific architecture:

make defconfig ARCH=x86_64

With the linux source configured, time to build! On a somewhat beefy desktop (48 threads, 128GB of RAM, Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz), this is how long a fresh build took:

$> time make -j $(nproc) 
Kernel: arch/x86/boot/bzImage is ready  (#1)

real    1m27.860s
user    45m30.886s
sys     4m37.614s

Yay, we built something!

Side note: finding the default makefile target

The default Makefile target (the target built when you run make with no arguments) is the first target you see after Updating goal targets when you run make -d:

make -d | grep "goal targets" -A 2 | head -n 10

Adding Debug symbols, KASAN

We’ve successfully built the linux kernel, but we don’t have any debug symbols. Since my usual purpose in working with the linux kernel is to debug, develop, or test some security idea with it, I also want KASAN enabled for the build.

This is done through the .config file that we created earlier. In make menuconfig, you can search for options using the / key.

KASAN-related options are in Main menu -> Kernel Hacking -> Memory Debugging secton.

Debug-info-related options (searching for DEBUG_INFO) are in the Kernel hacking -> Compile-time checks and compiler options -> Debug information -> Generate DWARF 5 Debug Info section.

Alternatively, individual options can be set with the scripts/config command. This can be rather tedious though. If you compare the default config with the resulting config after setting the KASAN and DEBUG_INFO_DWARF5 options through make menuconfig, you’ll find that there are many other changes to the config that resulted from setting those two options.

scripts/config -e KASAN -e DEBUG_INFO_DWARF5 # ... and add the rest of the settings

You should notice a change in the output of the file command when inspecting the vmlinux binary after rebuilding:

# before
$> file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=f90935181a31e429b7d27b284e1653783d4f12a9, not stripped

# after
$> file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=20312a83d39426267a334abd28f518017d9b64c4, with debug_info, not stripped

Running & Debugging the Linux Kernel

TL;DR

sudo mkinitramfs -o initfs

# in one shell - use CTRL+A,x to exit
qemu-system-x86_64 \
    -kernel arch/x86_64/boot/bzImage \
    -nographic \
    -append "console=ttyS0" \
    -m 1024 \
    -initrd initfs \
    --enable-kvm \
    -cpu host \
    -s -S \
    -fsdev local,path=$(pwd),security_model=none,id=test_dev \
    -device virtio-9p,fsdev=test_dev,mount_tag=test_mount

# in another shell
gdb vimlinux -ex "target remote :1234" -ex "c"

QEMU can be used to run the linux kernel with qemu-system-x86_64. If you’re cross compiling for different architectures, qemu can also be used to run those.

The main build output that we’ll be booting into is found in arch/x86_64/boot/bzImage:

$> file arch/x86/boot/bzImage 
arch/x86/boot/bzImage: Linux kernel x86 boot executable bzImage, version 5.18.0-rc7-g210e04ff7681 (guts@ungeheuer) #1 SMP PREEMPT_DYNAMIC Wed May 18 06:42:46 PDT 2022, RO-rootFS, swap_dev 0xA, Normal VGA

Skipping ahead to all of the options needed for qemu-system-x86_64 to run the linux boot image:

`qemu-system-...` Option	Notes
`-kernel`	Path to the kernel bzImage
`-nographic`	Do not start a GUI for this VM
`-append "console=ttyS0"`	Needed to have console output from the kernel
`-m 1024`	Megabytes of memory to give the VM (will error if too low)
`-initrd initfs`	The init filesystem used (will error if not present)
`--enable-kvm`	SPEED
`-cpu host`	Use a KVM processor with all of the supported host features
`-s`	Shorthand for `-gdb tcp::1234`
`-S`	Freeze the CPU at startup - need to continue in gdb
`-fsdev ...`
`-device virtio-9p,..`	This and the previous config setup host <-> guest sharing

The initfs file used with the -initrd option can be created with the mkinitramfs -o initfs command. I have to use sudo to create this on my machine:

$> sudo mkinitramfs -o initfs
$> file initfs
initfs: ASCII cpio archive (SVR4 with no CRC)

Notice that this is a cpio archive. You can use the cpio command to build your own root filesystem from a directory that you manually created.

The full command:

qemu-system-x86_64 \
    -kernel arch/x86_64/boot/bzImage \
    -nographic \
    -append "console=ttyS0" \
    -m 1024 \
    -initrd initfs \
    --enable-kvm \
    -cpu host \
    -s -S \
    -fsdev local,path=$(pwd),security_model=none,id=test_dev \
    -device virtio-9p,fsdev=test_dev,mount_tag=test_mount

Once this runs, you should see nothing - it’s waiting for gdb to attach and tell it to continue (because of the -s and -S flags).

In a separate shell, run:

$> gdb vmlinux
...
Reading symbols from vmlinux...
(gdb) target remote :1234
Remote debugging using :1234
0x000000000000fff0 in gdt_page ()
(gdb) c

This will attach us to the qemu-system-x86_64 process so that we can debug it! After telling gdb to continue (c), you should see the kernel booting up and then drop into a basic busybox shell:

[    1.397467] Run /init as init process
Loading, please wait...
Starting version 246.6-1ubuntu1.7
[    1.547386] e1000 0000:00:03.0 enp0s3: renamed from eth0
[    1.957531] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
[    2.884022] btrfs (169) used greatest stack depth: 13984 bytes left
done.
No root device specified. Boot arguments must include a root= parameter.[    2.893655] random: fast init done



BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu9.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)