27 April 2013

399. Looking at speeding up (re)boot on debian wheezy.

I'd be interested in getting my beowulf cluster nodes to boot a little bit faster -- (re)boots of the nodes very are infrequent, but the front node doubles as my work desktop and is normally rebooted at least once per month (kernel upgrades etc.) -- rebooting the front node makes me nervous, however, and the faster it boots, the better it is.

I should probably build a low-powered front node specifically for my cluster though...but that takes money, and money takes time.

Anyway, boot. In spite of the impetus for this post I'm testing this on my laptop which has wheezy, gnome 3.4 and an SSD -- it's not that representative of the target system and I'll have to repeat this on a normal desktop with a spinning hdd at a later stage.

I'm more or less following http://wiki.debian.org/BootProcessSpeedup. Note that insserv seems to be set up and enabled by default in Wheezy.

Timing it -- Setting up bootchart2
I first tried to define boot times arbitrarily as the time from me hitting enter in GRUB, to the visual appearance of the log-in prompt in GDM3, but it was too imprecise (up to +- 2) relative to the time a boot took (ca 9-10s).

I ended up installing bootchart and bootchart-view instead.
sudo apt-get install bootchart2

Then edit /etc/default/grub as shown here:
GRUB_CMDLINE_LINUX_DEFAULT="quiet initcall_debug printk.time=y init=/sbin/bootchartd"
and run
sudo update-grub

After a boot, run
eog bootchart.png

You'll get something like this:
Look at the top, right above the first chart -- it says 'time: 6.61s'. I'll use that as the metric.

Most of the time bootchart2 worked fine, but for the odd boot the /var/log/bootchart.tgz wasn't accepted by pybootchartgui.

Normal boot, pre-optimisation: 
'Cold' reboots: 6.61, 5.77 seconds
Warm* reboots: 6.46, 5.79, 5.97 seconds

*using shutdown -r now

The variability is very high -- there's almost a second between the fastest and slowest boots. Keep that in mind when looking at the numbers later on.

Using readahead-fedora to pre-load files
sudo apt-get install readahead-fedora

After install, readhead-early, -late and stop were enabled in rcconf.

The first boot took over 7 seconds, but later boots were typically around 6 seconds or faster. Note that readahead is solving an issue which isn't really present when using high bandwidth SSDs, and may even slow things down under conditions where you use an SSD or a spinning disk with a high rpm (e.g. >7200 rpm)

First run

'normal' run

Not exactly an improvement. Looking at /etc/readahead.d/custom.early shows that the wrong kernel files are loaded -- I'm using a custom kernel (3.8.5-ck1) but the stock kernel files are loaded (3.2.0-4). I edited custom.early to point towards my current kernel, and then did a warm reboot.

Speeding up reboots -Kexec
sudo apt-get install kexec-tools

Shutdown your computer once, then boot up. After that first time you can do warm reboots (sudo shutdown -r now) without going through the BIOS and grub stages. The only -- visible -- downside is that your screen will go crazy for a few seconds as the running kernel is being overwritten by the new kernel (I presume). Doesn't look pretty, but reboot is fast.

I couldn't get bootchart to time the hot reboots, but they look 'fast'.

I'll be repeating this on a system with a spinning disk at a later stage.


  1. I'd say this is all a big waste of time. The problem is kernel that is full of crud. A tight, customized for hardware, kernel takes a minute or two to compile and ~1.5s to boot ( that is latest dmesg message is:
    [ 1.541493] tsc: Refined TSC clocksource calibration: 3300.022 MHz
    [ 1.541498] Switching to clocksource tsc

    And after that it takes ~3-4s to boot debian. All that is on 7200RPM Sata disk.

    1. I can't say that I disagree -- part of the problem is that I didn't have access to a good system to do this on (I presume readahead might work better on a 5400 rpm drive), but whittling down the number of services that start, and having a leaner, hardware specific kernel should have a more substantial effect.

      On the other hand, tuning the kernel involves a fair amount of effort and time and takes a bit more expertise -- readahead and kexec requires nothing more than a successful installation.

      It was worth exploring, and while it didn't pay off, the investment was minimal.

    2. Not a waste of time at all since Lindqvist was nice enough to share his results, including steps to replicate, with everyone. He's probably saved a thousand people 30 minutes of time doing the same experiment.