Shellshock and Arch Linux

I’m guessing most people have heard about the security issue that was discovered in bash earlier in the week, which has been nicknamed Shellshock. Most of the details are covered elsewhere, so I thought I would post a little about the handling of the issue in Arch.

I am the Arch Linux contact on the restricted oss-securty mailing list. On Monday (at least in my timezone…), there was a message saying that a significant security issue in bash would be announced on Wednesday. I let the Arch bash maintainer and he got some details.

This bug was CVE-2014-6271. You can test if you are vulnerable by running

x="() { :; }; echo x" bash -c :

If your terminal prints “x“, then you are vulnerable. This is actually more simple to understand than it appears… First we define a function x() which just runs “:“, which does nothing. After the function is a semicolon, followed by a simple echo statement – this is a bit strange, but there is nothing stopping us from doing that. Then this whole function/echo bit is exported as an environmental variable to the bash shell call. When bash loads, it notices a function in the environment and evaluates it. But we have “hidden” the echo statement in that environmental variable and it gets evaluated too… Oops!

The announcement of CVE-2014-6271 was made at 2014-09-24 14:00 UTC. Two minutes and five seconds later, the fix was committed to the Arch Linux [testing] repository, where it was tested for a solid 25 minutes before releasing into our main repositories.

About seven hours later, it was noticed that the fix was incomplete. The simplified version of this breakage is

X='() { function a a>\' bash -c echo

This creates a file named “echo” in the directory where it was run. To track the incomplete fix, CVE-2014-7169 was assigned. A patch was posted by the upstream bash developer to fix this issue on the Wednesday, but not released on the bash ftp site for over 24 hours.

With a second issue discovered so quickly, it is important not to take an overly reactive approach to updating as you run the risk of introducing even worse issues (despite repeated bug reports and panic in the forum and IRC channels). While waiting for the dust to settle, there was another patch posted by Florian Weimer (from Red Hat). This is not a direct fix for any vulnerability (however see below), but rather a hardening patch that attempts to limit potential issues importing functions from the environment. During this time there was also patches posted that disabled importing functions from the environment all together, but this is probably an over reaction.

About 36 hours after the first bug fix, packages were released for Arch Linux that fixed the second CVE and included the hardening patch (which upstream appears to be adopting with minor changes). There were also two other more minor issues found during all of this that were fixed as well – CVE-2014-7186 and CVE-2014-7187.

And that is the end of the story… Even the mystery CVE-2014-6277 is apparently covered by the unofficial hardening patch that has been applied. You can test you bash install using the bashcheck script. On Arch Linux, make sure you have bash>=4.3.026-1 installed.

And because this has been a fun week, upgrade NSS too!

Who Packaged for Arch Linux in 2013

I was having a look at the current state of the Arch Linux repositories today in terms of the number of packages each person maintains and thought it interesting to see who did the packaging last year. So here are some numbers!

Firstly, the real repos (i.e. the repositories that TUs can not touch!). Note that the y-axis in this plot is the number of commits made to the repos and not the number of packages updated. Updating a package generally takes two commits and additional commits are done every time a package moves between repositories (e.g. moving packages out of [testing])

First, the two most prolific committers are Andrea Scarpino and Sven-Hendrik Haase. They both package KDE, which is in itself a lot of packages, but they get bonus commits for the [kde-unstable] repository where a lot of beta and release candidates are packaged. There is a lot of scripting going on for those rebuilds too, so don’t give them too much credit! Sven also deals with boost lately and the required rebuilds.

In places 3 and 6 are Jan Steffens and Jan de Groot who do our GNOME packaging. Rounding out the major desktop packagers is Evangelos Foutras in 9th place. In 4th, we have Andreas Radke who packages Xorg among other things including LibreOffice.

Eric Bélanger takes 5th place. I think he needs a specific shout out here because of all the effort he puts into maintaining the packages that officially have no maintainer. He regularly updates these packages and fixes their bugs. He also does far more than his share during rebuilds.

I am in 7th. This appears due to rebuilds for the removal /sbin et al., and the static libraries removal I pushed this year. In 8th is Tobias Powalowski who maintains the kernel package and deals with most of the module rebuilds.

Now a quick look at TU controlled repository commits. This includes the [community] and [multilib] repositories.

Not surprisingly, in first place we have Sergej Pupykin. He maintains about a 1/3 of the packages in the [community], although only has 1/6 of the commits… In 2nd place is Alexander Rødseth, who as far as I can tell does not maintain any specific package groups, so is just working hard! Bartłomiej Piotrowski is in 3rd (and who also rounded out the top 10 for the main repos) and we see Sven-Hendrik Haase again in 4th.

I’d also like to note the importance of all (or at least some!) of the people who have relatively few commits. In fact, I think we need more of them. I’d like to see the [extra] repo be almost exclusively the big package groups (Xorg, KDE, GNOME, XFCE, perl, python, etc) and [community] be all the additional packages with many more packagers each being responsible for a handful of packages that they are really interested in. So if you are thinking of applying for a Trusted User position, look at the tail of the distribution and do not let the big packagers put you off.

Shout-Out To Me on LAS (Not Really…)

For those that have not seen it, this week the Linux Action Show did a a wrap-up of their week long Arch challenge. If you are interested in that bit, start here… With praise such as “it is really not that bad”, I guess Arch Linux is doing well!

I got two indirect shout-outs. Clearly I am the developer who has an “ounce of contempt“. I live in the metric world, so that is 28.3495g, which seems relatively little. I also get a shout-out in their IRC screen! And to be clear, I am not a “my way is always right type”, it just works out that way.

Ways To Contribute To Arch Linux

I have been asked quite often recently “what is some ways I can contribute to Arch Linux?”. And the usual answers apply – package for the AUR, contribute to the wiki, help out on the forums and IRC… All useful to the whole community, but being not selfless at all I really want people to help in a way that helps me! So here are some ideas:

#1 – Fix Bugs on the Bug Tracker

This is not a new suggestion either… but it is an area that I think most people could contribute. I would say that there is some things that can be done to reduce the work for the developer that they are assigned to for the majority of bug reports in our tracker. For example:

  • Check that you can (still) replicate the bug. If you can, do not post “ME TOO!” as a comment. Just add a vote. Unless of course you can provide more information.
  • Summarize links to external reports. I hate bugs that are reported with just a link to a forum post or another bug tracker that requires me to read pages of posts to get at the issue. A brief summary of how to replicate is very useful.
  • For upstream bugs, make sure they are reported upstream. Provide the link to the upstream bug report or mailing list, etc.
  • If upstream has a patch for the issue, test it. If it works, provide the updated PKGBUILD and patch in the bug tracker, making sure a there is a comment in the PKGBUILD about where the patch came from.

#2 – Ensuring Continuously Buildable Arch

Arch rolls along a breakneck speed and quite often a package that was built a few weeks ago will no longer build from ABS. Then we get bug reports in the form “foo does not build”. In fact, most of those bug reports do not even provide build error (stupid users…)! These bugs particularly annoy me because I really dislike ABS in its current form. However, having packages that always build is very helpful when we do rebuilds due to library soname bumps or some change in packaging policy.

What I would propose is forming a group of people who in some way are continuously rebuilding Arch packages. That should be easy to script with the use of our devtools scripts. Any time a package no longer builds, the issue is investigated, reported upstream (if needed) and a fixed PKGBUILD is posted on the bug tracker. That way the developer only needs to quickly scan the updated PKGBUILD, confirm the fix is good, and push the rebuild to the repos.

#3 – Forming an Arch Security Team

Currently, Arch has a rather ad-hoc approach of dealing with security issues. This is because the majority of security fixes result in a new version of software being released, so we get them quickly with an update anyway. Also, developers tend to follow the mailing lists of the software they maintain. However, Arch could benefit from team who keep track of new security issues and make sure our packages get fixed. It would be worth doing a review of the current package situation too…

There was a project called “Arch Sheriff” a while ago that attempted to do this. However, I think it failed for two main reasons. It required the developers to go to an external website to check if their packages were vulnerable and, to a much lesser extent, it required them to go find the upstream fixes. A much better approach would be for the security team to interact with the developers through the bug tracker, providing fixed PKGBUILDs that include links to the issue and the upstream fix as comments.

#4 – Buy Allan Beer

On second thoughts, that probably hinders rather than helps…

For ideas #2 and #3, there will be some groundwork needed to get this underway. I guess using IRC, the wiki and mailing-lists should be enough for organizing how they will be run. I would not be directly involved in any of those ideas (they are suppose to free up my time!) but can advise and provide a middle man to the developers if needed. Continuously building packages for idea #2 would also require computing resources, but I guess that could be spread across people in the group.

So what are you waiting for? Get doing stuff!

Death of the [allanbrokeit] repository

I have deleted the [allanbroke] repository. It was started mainly to test the PGP signing implementation in pacman, which is now well established. Also, I would delay any packaging of release candidates or beta releases for this repository until I had enough free time and often official releases were made before that happened.

The repository may return someday, perhaps with VCS builds of packages I use locally once I get around to automating their creation as that would require no extra work…

My Arch Linux Talk at SINFO XX

I was recently invited to give a talk about Arch Linux at SINFO XX at IST in Lisbon, Portugal. It was a whirlwind tour of Europe, with the time I spent in transit almost exactly equal to the time I spent there.

Check out the video of my talk on their YouTube channel. I discuss what makes Arch different from other Linux distributions, what our strengths are as a distribution and briefly cover what future plans people have. I’m not going to watch it as it is never a good idea to dissect talks too much, so I’ll just assume I was awesome… It was also after midnight in my time zone, so I blame any mistakes on that.

Thanks to the organizers for inviting me over!

Edit: Quite a few people have asked for copies of me slides. Here they are (CC BY-SA): ODP PDF

Battle of the Arch Spin-Offs

According to Distrowatch, there are eleven “distributions” “based” on Arch Linux. I use “distributions” in quotes, because some are far less of a distribution than others and “based” gets quotes because some are so based that they are really Arch Linux in a poor disguise…

I have seen a bunch of new release announcements for several of these distributions in the last few days, so I thought I would take some for a spin and see what I am missing out on.

All distributions were tested in VirtualBox with a 128GB disk and 2GB RAM.

Contender #1: Chakra – 2012.12 Claire

The Chakra Project is my #1 contender as it is a real Linux distribution. They do use our PKGBUILDs (sometimes too directly…) but they rebuild everything for themselves. Chakra also has quite an interesting take on the rolling release model, opting for a “stable” core of the distribution and rolling release userland. So toolchain updates are rare, but the newest KDE will be packaged within days. I have issues with their “stable” core, because stable means no updates at all. Even minor bug fix updates are not considered and I am sure there is no backporting of any security fixes either as their team is too small to handle that. For example, I would not be running pacman-4.0.2 or even an unpatched pacman-4.0.3 as my distributions package manager. Another thing unique to Chakra is the pure KDE set-up. Any popular software that is not Qt based (e.g. firefox, GIMP) is supplied in a bundle. I find these a very bad idea, as many of the same libraries are on your system multiple times but in separate bundles, and it also appears that these are easy to break and under-maintained.

Onto the installer… Chakra wins points for being the only pure graphical installer out of the distributions I consider here. I did a review of the installer early in 2010 and there seemed a complete lack of improvement. The Live DVD booted fine and welcomed me with a window that pointed out the difference between Chakra and Arch, and noted that pacman is not designed for a GUI frontend (debatable…) so it will be replaced one day. Starting the install gives three screens worth of install notes which no-one has ever read and if anyone did read them, I am confident they did not care that their website is being redesigned. The partitioning took me to an external tool (KDE Partition Manager), which I actually found rather difficult to use. It took me several attempts to get everything set so that the installer would let me continue.

Once I finished the install, I rebooted… to a prompt. Whatever autodetection is done to get the KDE session up a running post-install had failed in VirtualBox. So no further review was done.

Contender #2: CinnArch – 2012.12.21

I had heard good things about the Cinnamon desktop, so CinnArch was my next choice. This distribution is Arch Linux – when I look at the package information for glibc, I see that I packaged it. The only difference is that an additional repo is provided with the packages needed for Cinnamon desktop to run.

The installer boots to a nice live system. There is two installation options – “CLI installer” and “Graphical Installer” but the latter is not selectable so “CLI” it is… In this case, CLI means an ascii-graphical menu driven installation environment that was easy to follow. The only steps that I found amusing was finding the fastest mirror to install from, which (with only slight exaggeration) took longer to complete than downloading the packages from the slowest mirror would have. And then on with installing the packages. Or not… It turns out that the installer detects I am using VirtualBox and tries to install the virtualbox-guest-modules package, which currently has a broken dependency and so can not be installed. There is no way to skip this that I could find.

So lets play around in the live media to see what might have been. It turns out that CinnArch provides two repos: cinnarch-repo and cinnarch-core. I could not find the justification for requiring a second repo. The repo looks fairly good on a quick glance, with all packages signed and a keyring provided. Theoretically, you can just enable that repo and install cinnamon from there, but I could not find where to see a list of keys that the developers use that would enable me to install the keyring package and then verify the rest of the packages. One thing I was concerned about was the filesystem package that was being provided for reasons I could not ascertain. The biggest issue with providing a repo for Arch Linux is that you need to keep up with our rolling base. That means your repo needs to have rebuilds available when we push (e.g.) a library update with an soname change. Providing an unnecessary filesystem package seems to only make things difficult for them. This was also my first time seeing the Pacman-XG GUI for the pacman package manager. That is enough said about that…

Contender #3: Manjaro – 0.8.3 XFCE edition

I was not expecting much from Manjaro given I had read this review of a previous installer. But given the “success” of the last two distros, it has a chance… Some things from that review had not changed – in the live environment there is still the unnecessary password of “manjaro” for both the user and root. And I was not endured by the green colour scheme or the logo which is used for the “start” menu. But to give Manjaro credit, I successfully installed using the old style Arch ascii-graphical install, which was easy for those that are use to that. Reiser3 as the default filesystem still confuses me.

So what do you get from Manjaro? It seems to be Arch delayed plus extras. The idea is that the Arch repos are delayed reaching you for a few days while everything is sorted out to give you an easy upgrade. This happens by adding a package manjaro-system which is always upgraded first and runs a script that automatically handles any manual intervention that would be required on a pure Arch Linux system. As a heads up, this uses a feature of pacman called SyncFirst that is removed for the upcoming pacman-4.1, so they may need to rethink their entire system soon.

The first difference I noted was it ran a 3.4.24 kernel, so that have held that back from being updated to the newest release, but at least it is the newest 3.4 series release. Looking at the glibc package once again, I am the packager, but it is a repo called [platform]. The Arch [extra] and [community] repos appear to be used as-is, so the need for [basis], [platform] and [addon] repos to replace our small [core] repo is strange. Also, this was the first time I had ever heard of a GUI for pacman with the imaginative name pacman-gui. It allows you to run pacman -Sy, which just refreshes the databases – a crazy thing to do without immediately updating in a rolling release distribution that does not provide multiple versions of a library. And it is not much of a GUI either, as it just launches a terminal that runs the pacman command you just clicked. There was also a LibreOffice installer that called pacman in the background.

The Winner?

Sadly, I do not think there was any winners today. Chakra had the most polished live environment, but failed to boot to the desktop. CinnArch failed through no fault of its own, which is sort of a fault of its own for not being a real distribution. Manjaro installed, although I saw nothing to make me want to recommend it.

The may be the most painful statement I have ever made… but ArchBang might be the winner!

Edit: Twelve hours later and the bug on the Arch end preventing the CinnArch install has been fixed. CinnArch installed and booted to the desktop without an issue.

Unmoderated Mailing List For Arch Discussion

It seems I have gained a “fan” or two over the last week. But this guy is my current favourite. Shutting down the arch-general mailing list for a day resulted in me getting the following email:

Subject: wrong move
Date: Wed, 26 Sep 2012 15:16:03 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Very bad move .

You will just inflame it for later if you have not learnt that yet what
are you doing running a mail list .. ?..

Pete .

Thanks Pete. Lets see who gets angry at me the most…

Subject: List admin
Date: Thu, 27 Sep 2012 10:42:44 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: arch-general-owner@archlinux.org

I wish to know who is the main arch-general@archlinux.org admin

i wish to start a vote to remove Allan McRae
or at least demote him . He appears to be prone to panic attacks and
ill judged moves ie shutting the list down ..

Before anyone says anything else i run 4 lists myself , all motor sport
related .

The other point is if certain people would just answer the question
instead of going off on some tangent it would help lessen the wars

You may not agree that is not my problem i see what i see this list
needs someone that does NOT panic

Pete

It would be a fair assumption that if I have the ability to shut down an email list, then I might have administrator access… A lack of reply and this email arrives:

Subject: Mail owner
Date: Mon, 1 Oct 2012 08:26:01 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: arch-general-owner@archlinux.org

I am once again calling for the removal of Allan McRae as an
unsuitable person to be in control of a mailing list ,

He is Childish despotic offensive officious obnoxious and is
stomping around like some little Hitler .

He has go to GO

Pete

The use of “despotic” was particular non-inventive given that was already used on the arch-general mailing list earlier that day. So the Godwinism was the best that could be done.

I subtly mentioned these emails in a reply to arch-general in order to let him know that I have seen them. That resulted in:

Subject: Un warrented transference of mails to list
Date: Mon, 1 Oct 2012 19:07:26 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Excuse me .

I kept OFF the list for a reason , you are in a flat out panic spin
this shows a mile off by your actions .

If you are unwilling to run the list and let it run then i suggest
you retire and let someone else do the job .

You are getting as bad if not worse than some of the bird brains on
the suse lists that is the reason a good number of new users have
appeared here Like me they were getting sick of the blatant junk from
the list owners .

This mail was sent OFF list for a reason get with it .

Oh and i still say RETIRE if you cant handle it ..

Pete

If you did not want emails to be public, perhaps you should not have sent them to an email address that you had no idea who the recipient was. Or, even better, just do not send them at all. I actually made an exception to not answering any of these emails and replied with that friendly piece of advise.

But that was not the end of it…

Subject: tosser
Date: Wed, 3 Oct 2012 08:53:15 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Allan

you are by far the biggest silly little boy i have ever had the
misfortune to stumble on.

How the Arch Linux community ever managed to get a twat like you as
the mailing list owner i will never know but hey there will soon be a
new mailing list .

Ban my aunt fanny word of warning WATCH your list you
think you have stopped me THINK AGAIN :-) … Oh and IP filtering
will get you nowhere at all

have fun

Pete .

I feel sad for his inability to form good insults. Using “biggest” and “little” together shows a poor grasp of the English language. This would be fine for non-native speakers, but from his email address I conclude that he is in the UK and given there is a person who dislikes immigrants with the same name on Google+ (who has me in a circle…), I will assume he is a native English speaker too.

Which finally gets me to the point of this post. It appears Pete will soon be creating a mailing list where people can rant about Arch Linux and the direction it is moving all they want with no fear of being censored. Thanks Pete!

Replacing Systemd In Arch Linux

For those who came here looking for the solution to the systemd “problem” in Arch Linux, this is not the article you are looking for. I care very little about my init system beyond that it should successfully boot my computer and start the software I need it to start. In fact, my entire understanding of the boot process goes “Push button… *MAGIC* …prompt”, so taking advice from me on the boot-up process may not be the best idea…

However, what I do know about is the Arch Linux packaging system and how to put together a Linux distribution. So I am going to discuss how people would go about providing all the tools to run an init system that is not systemd in Arch Linux. Many lessons will be taken from how systemd was provided in Arch Linux; firstly as a community based projected and then as official packages providing a secondary init choice in the repos. From what I see from current efforts, people seem slightly naive about what is required and are completely ignoring what has gone previous.

Lets start with choice number one. Where are you going to get udev from? I see two choices here. Firstly, you could just create a package containing on the udev files from the systemd tarball. This is the approach used by Linux From Scratch and you could even use their Makefile. The original email announcing the merge of udev into systemd states that udev can still be built for usage outside of systemd systems and that will be supported officially. So I personally would choose that option. However, I know people are concerned that udev will become more fully dependent on systemd. Here is the email that people cite as the end on non-systemd udev. I read that as saying that genuine issues when using udev outside of systemd will be fixed. There is also nothing saying that patches for udev that do not impact its usage in systemd will not be applied. Anyway, for those people there is a fork of udev being developed. If you select this option, there are a couple of things to be aware of. Firstly, there is is no guarantee of compatibility with the udev from systemd so the libudev. In fact, the fork has a different soname and that means you will need to recompile all software that links to it (~30 packages in the Arch repos). Secondly, the development of this fork currently appears to be porting relevant commits from the systemd tree to a snapshot of the udev codebase before the merge happened. It will be of interest to see what happens as these code-bases diverge and whether any independent development (excluding the build system…) occurs in the fork. (See Edit #2 below)

Choice number two is what you are going to do with software that links to the various libsystemd-* libraries. The anti-systemd way is to rebuild all this software to not have this dependency. That is only 12 packages at the moment, although the number is growing… An approach requiring no rebuilding would to just provide a libsystemd package. These are just libraries on your system and most software that uses them has something like if(sd_booted()) in it. That results in these doing nothing when your system was not booted with systemd.

If providing separate packages for udev and libsystemd, you will need to be careful with the provides array in their PKGBUILDs so that you avoid unnecessary rebuilds. But speaking of unnecessary… Why provide separate packages for udev and libsystemd at all? Just have the systemd package installed. If you do not boot with systemd, then its binaries just sit around on your system doing nothing but take a whopping 10MB of space (that number is pure guesswork…). Call me lazy, but all that packaging seems a lot of effort. Do you repackage libjpeg-turbo to get rid of its binaries that you do not use?

Once you have managed to get that far, you will need to figure out which init system you will use and how you will manage services. I am going to state that the current Arch Linux initscripts are a dead end. The current version requires systemd binaries, so to avoid that you would need to grab an old release. Then you would have to fix the bugs that were fixed by moving to the use of systemd binaries. Finally, when Arch Linux moves to systemd by default, the service management scripts in /etc/rc.d will gradually be removed, which is no great loss as they are horrid anyway. I will also assume that some monitoring of services and restarting them when they fail is a worthwhile goal, as no-one really has spoken out against that. Completely ignoring upstart (because yuck…), you have two real options for this.

Option #1 is to use runit. This is a complete init system replacement with service management and seemed to be a popular choice among people with severe systemd allergies on the arch-general mailing list. As a bonus, the website has a whole heap of run scripts that you can use to provide a runit-arch-services package (analogous to the systemd-arch-units package that was provided in the early systemd days and is currently being phased out). Note that means you are going to have to provide an runit run file for every piece of software in the Arch repos if you are doing a proper job. If you are going to use runit, you may want to also consider the ignite project which provides additions to runit that allows it to use some old style /etc/rc.conf style configuration. I am going to pretend that ignite‘s definition of “adapted from Arch initscripts” does not mean GPL copyrighted code was copy-pasted and released as public domain!
(See Edit #1 below)

Option #2 is to use OpenRC, which is developed by people over at Gentoo. This works with your systems init, so you are going to need to keep sysvinit as well. Because it is compatible with the Gentoo init scripts, you will probably be able to find the daemon scripts you need in the Gentoo portage tree. I assume for a more “old school” Arch Linux experience, you would want to do something similar to ignite and pilfer relevant bits from the Arch Linux initscripts package. I believe on of the openrc developers posted to the arch-general mailing list suggest it was used, so there is probably help available getting this running for those that are interested.

That is a brief summary of what is required to purge your system of the “systemd virus” – for now. But as a virus, it will spread… Soon you will need to deal with a lack of ConsoleKit and if all the hysteria is to be believed, every single other piece of software on your system will soon be absorbed into systemd. So if you are going to do a lot of work to avoid systemd, be prepared for the amount of work to increase in the future. But do not concern yourself too much… Arch Bang has plans to save all the systemd haters! And they have made an installer so their ability to handle actual packaging of something this complex is not to be questioned!

Post publication edits:
Edit #1: I am removing my comment about the licensing of ignite for two reasons. 1) The author contacted me and pointed out the only directly copied bits was the mountpoint code (~4 lines). I see other similarities, but admittedly there is not many other ways to do [[ $foo ]] && ... style tests. 2) There is no license in the Arch Linux initscripts code base. It used to print “GPL2″ when booting, but that has even been removed.

Edit #2: Do you know how to fix the udev fork having a different soname for its library? Use this patch! Now I am convinced the author of the fork knows too little for it to be useful.

Switching My Laptop To Systemd

I hear rumours that Arch Linux is probably switching to systemd as its default init system at some stage. Or at least that is what has been reported on by some Linux “news” sites based on a post to the developmental mailing list proposing we start the migration. I personally would have waited for something more official before reporting, especially because there is a lot that needs done before that can happen.

Anyway… with that upcoming change, I decided it was time for me to finally install systemd and see how hard the transition would be. The answer? Not that hard at all, although I made it rather difficult for myself as you will read.

I figured the first step in moving towards using systemd was to deal all the configuration files. I tend to ignore updating configuration files unless something stops working, so my /etc/rc.conf was full of LOTS of old settings. Following the archlinux(7) man page, I altered /etc/hostname, /etc/locale.conf, /etc/vconsole.conf and /etc/timezone and removed all their parts from rc.conf. I also deleted the network section (which has been useless forever given I use NetworkManager) and the modules section (as the modules are all autoloaded these days). That left the DAEMONS line. I rebooted to see whether everything was still OK at this stage and I struck failure #1. The console went all funny during boot-up, changing characters into weird symbols and losing colour. So my vconsole.conf was obviously bad. From its man page: “FONT= defaults to latarcyrheb-sun16“. What I did not realize was that means you still have to add a FONT= line.

Now to actually install systemd. A simple pacman -S systemd, adding init=/bin/systemd to my grub kernel line and I was ready to go. I want my nice looking LXDM at boot-up, so I enabled that service with systemctl enable lxdm.service. And reboot!

Booting when fine, I logged in and everything was happy… which confused me. I had not enabled all the other services to replace the daemons I start in rc.conf. It turns out that some work has been done by the relevant Arch Linux developers to help people transition to systemd and it will still load modules and daemons specified in rc.conf. But I want a pure systemd boot-up so I need to disable that. Unfortunately, I could find no sensible way to achieve that. I could either comment out DAEMONS from rc.conf – but that makes reverting to using the old initscripts require more than altering the kernel line – or I could be heavy handed and delete the service files from the initscripts package. I did the latter.

So systemd booting attempt #2. Prior to rebooting, I enabled the NetworkManager service as I like having access to the internet while playing with my system. Reboot and all is fine. Sort of… I have internet access, so Network Manager is running, but where is my applet? I need that applet to easily access the range of networks my laptop goes on and VPNs etc. My strategy for solving things like this goes: #1 – run from a terminal and see if there is an error message; #2 – run as root and see if that works. Running nm-applet as root worked. I figured this was maybe LXDM not being completely systemd friendly yet and something not being launched properly, so I edited /etc/dbus-1/system.d/org.freedesktop.NetworkManager.conf and gave my user more permissions. And all is fine again…

Until I try getting sound. I was in the middle of watching Fight Club when I embarked on this adventure and despite having seen it before, sound is still important. Looking in the alsa package file list I see a couple of service files. I try starting them and I get a message: They are not meant to be enabled using systemctl. Lets try running alsamixer – nothing – and as root? It works… Poop.

Now remember how I said that I do not deal with configuration file updates. My system is littered with .pacnew files. Obviously I am missing permissions due to logging in with LXDM. And sure enough, /etc/pam.d/lxdm had an update and that involved adding pam_systemd.so. Sounds important. Fix that, revert my changes for NetworkManager and reboot. Internet and sound. Great success!

On to enabling all my other daemons. No need to deal with dbus or netfs as they are handled automatically and the new logging is fine by me so syslog-ng can go. I enable ntpd.service and then attempt to start crond.service… Fail. Looking in the cronie package, I see a cronie.service file that works. Strange thing is, there is a crond.service file and it is a symlink to the cronie.service. I have not idea why the symlinked version can not be enabled, but I strike the same thing again for cupds.

The only other daemon left is my custom fan control. For that, I need to write my own service file. It is a very simple daemon, so has a very simple service file:

[Unit]
Description=Allan's MacBook Pro Fan Control Daemon
 
[Service]
ExecStart=/usr/bin/mbpfan
 
[Install]
WantedBy=multi-user.target

Much better than the 70 line monstrosity for the the old rc.d script.

The final daemons I used to launch manually were php-fpm, mysqld and nginx to run WordPress locally when I want to draft a blog post. I heard that you could easily customize service files so I wanted to add a dependency on php-fpm and mysqld to nginx so I only need to launch one thing. That was as simple as creating /etc/systemd/system/nginx.service with:

.include /usr/lib/systemd/system/nginx.service
 
[Unit]
Requires=mysqld.service php-fpm.service

Everything was running, so now it is time to remove initscipts and sysvinit. And a final reboot to ensure everything is still fine… but I can not longer shut down from within XFCE. Of course, removing sysvinit gets rid of the shutdown binary, but that is replaced by installing systemd-sysvcompat. That also provides a /sbin/init symlink so I can remove the addition to my GRUB kernel line. Finally reboot and almost all is fine. I am a picky bastard and so could not accept there being some bootup output not cleared from my tty1. Adding “quiet” to the kernel line removes all that output and I am happy.

I have not been using systemd long enough to give it a proper review. But I can say it works, at least for me. Also, shut down is now amazingly fast. I know that may not sound that important, but I do shut down my laptop in a rush at the end of the work day to run for the train. So saving time there is good for me. There was essentially no difference in boot-up speed for me. But NetworkManager is now fully up and running and connected to the network by the time I get logged into XFCE now. It is also what takes the most time to launch, so I wonder if that is holding up the process somehow. The only thing that annoyed me was not being able to disable the rc.conf parsing services, but I am informed there is a patch for that on the arch-projects mailing list. That should make it a lot easier for people switch back and forward between using SysVinit and systemd when doing this transition.

PS. I know I capitalized systemd in the title when it should not be… but I like my post titles with capitals.