For those who came here looking for the solution to the systemd “problem” in Arch Linux, this is not the article you are looking for. I care very little about my init system beyond that it should successfully boot my computer and start the software I need it to start. In fact, my entire understanding of the boot process goes “Push button… *MAGIC* …prompt”, so taking advice from me on the boot-up process may not be the best idea…

However, what I do know about is the Arch Linux packaging system and how to put together a Linux distribution. So I am going to discuss how people would go about providing all the tools to run an init system that is not systemd in Arch Linux. Many lessons will be taken from how systemd was provided in Arch Linux; firstly as a community based projected and then as official packages providing a secondary init choice in the repos. From what I see from current efforts, people seem slightly naive about what is required and are completely ignoring what has gone previous.

Lets start with choice number one. Where are you going to get udev from? I see two choices here. Firstly, you could just create a package containing on the udev files from the systemd tarball. This is the approach used by Linux From Scratch and you could even use their Makefile. The original email announcing the merge of udev into systemd states that udev can still be built for usage outside of systemd systems and that will be supported officially. So I personally would choose that option. However, I know people are concerned that udev will become more fully dependent on systemd. Here is the email that people cite as the end on non-systemd udev. I read that as saying that genuine issues when using udev outside of systemd will be fixed. There is also nothing saying that patches for udev that do not impact its usage in systemd will not be applied. Anyway, for those people there is a fork of udev being developed. If you select this option, there are a couple of things to be aware of. Firstly, there is is no guarantee of compatibility with the udev from systemd so the libudev. In fact, the fork has a different soname and that means you will need to recompile all software that links to it (~30 packages in the Arch repos). Secondly, the development of this fork currently appears to be porting relevant commits from the systemd tree to a snapshot of the udev codebase before the merge happened. It will be of interest to see what happens as these code-bases diverge and whether any independent development (excluding the build system…) occurs in the fork. (See Edit #2 below)

Choice number two is what you are going to do with software that links to the various libsystemd-* libraries. The anti-systemd way is to rebuild all this software to not have this dependency. That is only 12 packages at the moment, although the number is growing… An approach requiring no rebuilding would to just provide a libsystemd package. These are just libraries on your system and most software that uses them has something like if(sd_booted()) in it. That results in these doing nothing when your system was not booted with systemd.

If providing separate packages for udev and libsystemd, you will need to be careful with the provides array in their PKGBUILDs so that you avoid unnecessary rebuilds. But speaking of unnecessary… Why provide separate packages for udev and libsystemd at all? Just have the systemd package installed. If you do not boot with systemd, then its binaries just sit around on your system doing nothing but take a whopping 10MB of space (that number is pure guesswork…). Call me lazy, but all that packaging seems a lot of effort. Do you repackage libjpeg-turbo to get rid of its binaries that you do not use?

Once you have managed to get that far, you will need to figure out which init system you will use and how you will manage services. I am going to state that the current Arch Linux initscripts are a dead end. The current version requires systemd binaries, so to avoid that you would need to grab an old release. Then you would have to fix the bugs that were fixed by moving to the use of systemd binaries. Finally, when Arch Linux moves to systemd by default, the service management scripts in /etc/rc.d will gradually be removed, which is no great loss as they are horrid anyway. I will also assume that some monitoring of services and restarting them when they fail is a worthwhile goal, as no-one really has spoken out against that. Completely ignoring upstart (because yuck…), you have two real options for this.

Option #1 is to use runit. This is a complete init system replacement with service management and seemed to be a popular choice among people with severe systemd allergies on the arch-general mailing list. As a bonus, the website has a whole heap of run scripts that you can use to provide a runit-arch-services package (analogous to the systemd-arch-units package that was provided in the early systemd days and is currently being phased out). Note that means you are going to have to provide an runit run file for every piece of software in the Arch repos if you are doing a proper job. If you are going to use runit, you may want to also consider the ignite project which provides additions to runit that allows it to use some old style /etc/rc.conf style configuration. I am going to pretend that ignite‘s definition of “adapted from Arch initscripts” does not mean GPL copyrighted code was copy-pasted and released as public domain!
(See Edit #1 below)

Option #2 is to use OpenRC, which is developed by people over at Gentoo. This works with your systems init, so you are going to need to keep sysvinit as well. Because it is compatible with the Gentoo init scripts, you will probably be able to find the daemon scripts you need in the Gentoo portage tree. I assume for a more “old school” Arch Linux experience, you would want to do something similar to ignite and pilfer relevant bits from the Arch Linux initscripts package. I believe on of the openrc developers posted to the arch-general mailing list suggest it was used, so there is probably help available getting this running for those that are interested.

That is a brief summary of what is required to purge your system of the “systemd virus” – for now. But as a virus, it will spread… Soon you will need to deal with a lack of ConsoleKit and if all the hysteria is to be believed, every single other piece of software on your system will soon be absorbed into systemd. So if you are going to do a lot of work to avoid systemd, be prepared for the amount of work to increase in the future. But do not concern yourself too much… Arch Bang has plans to save all the systemd haters! And they have made an installer so their ability to handle actual packaging of something this complex is not to be questioned!

Post publication edits:
Edit #1: I am removing my comment about the licensing of ignite for two reasons. 1) The author contacted me and pointed out the only directly copied bits was the mountpoint code (~4 lines). I see other similarities, but admittedly there is not many other ways to do [[ $foo ]] && ... style tests. 2) There is no license in the Arch Linux initscripts code base. It used to print “GPL2” when booting, but that has even been removed.

Edit #2: Do you know how to fix the udev fork having a different soname for its library? Use this patch! Now I am convinced the author of the fork knows too little for it to be useful.

I hear rumours that Arch Linux is probably switching to systemd as its default init system at some stage. Or at least that is what has been reported on by some Linux “news” sites based on a post to the developmental mailing list proposing we start the migration. I personally would have waited for something more official before reporting, especially because there is a lot that needs done before that can happen.

Anyway… with that upcoming change, I decided it was time for me to finally install systemd and see how hard the transition would be. The answer? Not that hard at all, although I made it rather difficult for myself as you will read.

I figured the first step in moving towards using systemd was to deal all the configuration files. I tend to ignore updating configuration files unless something stops working, so my /etc/rc.conf was full of LOTS of old settings. Following the archlinux(7) man page, I altered /etc/hostname, /etc/locale.conf, /etc/vconsole.conf and /etc/timezone and removed all their parts from rc.conf. I also deleted the network section (which has been useless forever given I use NetworkManager) and the modules section (as the modules are all autoloaded these days). That left the DAEMONS line. I rebooted to see whether everything was still OK at this stage and I struck failure #1. The console went all funny during boot-up, changing characters into weird symbols and losing colour. So my vconsole.conf was obviously bad. From its man page: “FONT= defaults to latarcyrheb-sun16“. What I did not realize was that means you still have to add a FONT= line.

Now to actually install systemd. A simple pacman -S systemd, adding init=/bin/systemd to my grub kernel line and I was ready to go. I want my nice looking LXDM at boot-up, so I enabled that service with systemctl enable lxdm.service. And reboot!

Booting when fine, I logged in and everything was happy… which confused me. I had not enabled all the other services to replace the daemons I start in rc.conf. It turns out that some work has been done by the relevant Arch Linux developers to help people transition to systemd and it will still load modules and daemons specified in rc.conf. But I want a pure systemd boot-up so I need to disable that. Unfortunately, I could find no sensible way to achieve that. I could either comment out DAEMONS from rc.conf – but that makes reverting to using the old initscripts require more than altering the kernel line – or I could be heavy handed and delete the service files from the initscripts package. I did the latter.

So systemd booting attempt #2. Prior to rebooting, I enabled the NetworkManager service as I like having access to the internet while playing with my system. Reboot and all is fine. Sort of… I have internet access, so Network Manager is running, but where is my applet? I need that applet to easily access the range of networks my laptop goes on and VPNs etc. My strategy for solving things like this goes: #1 – run from a terminal and see if there is an error message; #2 – run as root and see if that works. Running nm-applet as root worked. I figured this was maybe LXDM not being completely systemd friendly yet and something not being launched properly, so I edited /etc/dbus-1/system.d/org.freedesktop.NetworkManager.conf and gave my user more permissions. And all is fine again…

Until I try getting sound. I was in the middle of watching Fight Club when I embarked on this adventure and despite having seen it before, sound is still important. Looking in the alsa package file list I see a couple of service files. I try starting them and I get a message: They are not meant to be enabled using systemctl. Lets try running alsamixer – nothing – and as root? It works… Poop.

Now remember how I said that I do not deal with configuration file updates. My system is littered with .pacnew files. Obviously I am missing permissions due to logging in with LXDM. And sure enough, /etc/pam.d/lxdm had an update and that involved adding pam_systemd.so. Sounds important. Fix that, revert my changes for NetworkManager and reboot. Internet and sound. Great success!

On to enabling all my other daemons. No need to deal with dbus or netfs as they are handled automatically and the new logging is fine by me so syslog-ng can go. I enable ntpd.service and then attempt to start crond.service… Fail. Looking in the cronie package, I see a cronie.service file that works. Strange thing is, there is a crond.service file and it is a symlink to the cronie.service. I have not idea why the symlinked version can not be enabled, but I strike the same thing again for cupds.

The only other daemon left is my custom fan control. For that, I need to write my own service file. It is a very simple daemon, so has a very simple service file:

[Unit]
Description=Allan's MacBook Pro Fan Control Daemon
 
[Service]
ExecStart=/usr/bin/mbpfan
 
[Install]
WantedBy=multi-user.target


Much better than the 70 line monstrosity for the the old rc.d script.

The final daemons I used to launch manually were php-fpm, mysqld and nginx to run WordPress locally when I want to draft a blog post. I heard that you could easily customize service files so I wanted to add a dependency on php-fpm and mysqld to nginx so I only need to launch one thing. That was as simple as creating /etc/systemd/system/nginx.service with:

.include /usr/lib/systemd/system/nginx.service
 
[Unit]
Requires=mysqld.service php-fpm.service


Everything was running, so now it is time to remove initscipts and sysvinit. And a final reboot to ensure everything is still fine… but I can not longer shut down from within XFCE. Of course, removing sysvinit gets rid of the shutdown binary, but that is replaced by installing systemd-sysvcompat. That also provides a /sbin/init symlink so I can remove the addition to my GRUB kernel line. Finally reboot and almost all is fine. I am a picky bastard and so could not accept there being some bootup output not cleared from my tty1. Adding “quiet” to the kernel line removes all that output and I am happy.

I have not been using systemd long enough to give it a proper review. But I can say it works, at least for me. Also, shut down is now amazingly fast. I know that may not sound that important, but I do shut down my laptop in a rush at the end of the work day to run for the train. So saving time there is good for me. There was essentially no difference in boot-up speed for me. But NetworkManager is now fully up and running and connected to the network by the time I get logged into XFCE now. It is also what takes the most time to launch, so I wonder if that is holding up the process somehow. The only thing that annoyed me was not being able to disable the rc.conf parsing services, but I am informed there is a patch for that on the arch-projects mailing list. That should make it a lot easier for people switch back and forward between using SysVinit and systemd when doing this transition.

PS. I know I capitalized systemd in the title when it should not be… but I like my post titles with capitals.

With the recent “controversies” in Arch Linux – such as the move to /lib being a symlink, parts of the rc.conf file moving into separate files and the installation becoming command line based – I started thinking about what really defined Arch Linux in the early days and what make it what it is currently. To assess this, I read through all the news and interviews about Arch up until the 0.8 release. That covers about the first five years of the Arch Linux existence. Here is what I took from those articles, in no particular order:

Pacman
This is probably the most unique thing about Arch Linux and it is the reason I started using Arch in the first place. I may be ever so slightly biased here… but I still find this one of the best package managers around. This is not necessarily for the pacman manager itself, but the package building system is so incredibly simple. Let me be clear in pointing out that there are a lot of areas that I think could be substantially improved, but the major things I want implemented are gradually being crossed off my TODO list.

The Arch Build System (ABS)
Being able to readily customize packages is always touted as on of Arch’s best features in any review I see. I never use it… Well, not entirely true – I have 1 customized package out of the 626 currently on my system. I wonder if it is the packaging system rather than ABS that is being highlighted by not well informed reviewers there, because maintaining a custom package is really not a pleasant task in Arch.

Rolling release
Having packages available in our repos very quickly after the upstream release is a major attraction to Arch Linux. While Arch is far from the only rolling release distribution, the updates tend to be damn fast!

Optimized for i686
Ten years ago, most distributions were compiled for i486. So back then, compiling for i686 was a big feature. Now 64-bit has overtaken the world and and most 32-bit distribution offerings are built for i686, so this is not an advantage anymore. We should probably drop i686 and crank up the optimization of the x86_64 port!

Simple filesystem layout
I found this an interesting feature to be mentioned by Judd… Especially given all sorts of packages in Arch used to be shoved in /opt. But he was referring to the lack of things like /usr/doc. Years ago, Arch used to ship packages without documentation – including info pages. Now we do ship info pages (they are treated identically to man pages by the packaging system) and documentation is also shipped (in /usr/share/doc, not /usr/doc).

Simple configuration
The rc.conf file was highlighted by many reviews as being the one stop for all your configuration. In fact, one review suggested that it should also include “wi-fi profiles, printer, scanners, bluetooth etc” given there were not GUI tools to do this. The lack of distro specific tools for configuration was also highlighted, although that is a bit weird given our rc.conf is ditro specific.

Support
The help on the IRC channel, and to a lesser extent the forums, was often mentioned. Also the excellent documentation in the wiki was beginning to make its appearance.

Rapid adoption of new features
Having the latest an greatest software was often considered a highlight. For the initial five years of Arch Linux, that included things like the Ext3 and Reiser filesystems, udev support and Linux 2.6.

One thing I did not see mentioned anywhere was Arch’s use of vanilla packages. The minimal patching is one of my favorite features of Arch Linux, but I guess that is not something that would be readily seen by a reviewer.

So… lets take a look at the current controversies in Arch Linux land with respect to the foundations of Arch Linux.

Controversy #1 – Symlinking /lib to /usr/lib
People were not particularly happy about this. And they are really going to hate when /bin, /sbin and /usr/sbin all point to /usr/bin… Note there were quite a few other things that have become symlinks recently too (think about what points at the /run directory). The difficulties of the /lib change were exasperated by the rolling release model and limitations in pacman in performing the upgrade. But I think this fits in directly with the “simple filesystem layout” above. As discussed above, we have slowly moved away from this principle to provide packages closer to what their developers intended (by not removing documentation), so this is really a return to original Arch principles!

Controversy #2 – The demise of /etc/rc.conf
While the single rc.conf is highlighted as major feature of Arch Linux, reading the reviews makes you notice that configuration of an Arch install was never down to a single file. Other files mentioned included…

/etc/conf.d/wireless
/etc/fstab
/etc/group
/etc/grub.conf
/etc/hosts
/etc/inittab
/etc/lilo.conf
/etc/locale.gen
/etc/mkinitrd.conf
/etc/modules.conf
/etc/modprobe.conf
/etc/rc.local
/etc/resolv.conf
/etc/pacman.conf
/etc/X11/XF86config
~/.xinitrc

And that is just the configuration files mentioned in what were quite superficial reviews of Arch Linux. So a single configuration file was really just a myth.

But lets take a step back here… How about some quotes from Judd, the founder of Arch Linux:

“In Arch “simple” is different what other distros are considering. The learning is more important than getting something easily done.”

“Relying on GUIs to build/use your system is just going to hurt a user in the end. At some point in time a user will need to know all that some GUIs hide.”

What does a single configuration file get us? Is it “simple” in terms of the definition above? Not really… It requires that we write our own tools to deal with it and convert these into the appropriate commands during boot-up. For example, instead of configuring modules in /etc/rc.conf, we should directly use the configuration files that the module loading software reads. How is putting the settings in rc.conf and having software interpret them any different than using a GUI to do the same thing? It does not give us any actual understanding of the system. Also using the same configuration files as other distributions do (and this is becoming more standardized) means the learning done in Arch Linux will apply to many other distributions.

Controversy #3 – The move towards systemd
I think it is plainly obvious that systemd will become the primary system initialization method for Arch Linux in the not to distant future (not that this has been formally decided, so take that statement as unofficial…). Having not used systemd at all, I really can not comment directly on if it is any good or not. But what do we lose by moving to systemd in Arch? I have seen the following criticisms:

Users are being forced to use systemd. This argument really does not hold water. How many initialization systems are currently in use on Arch systems? Just the one currently “forced” on you…

The module control is more complex. Right… because “rc.d start foo” is much simplier than “systemctl start foo“. I think the real criticism people are trying to express is that the rc.d way of doing things is more traditional Arch. But have you looked at the quality of scripts in /etc/rc.d? It is generally poor. Moving to systemd unit files instead will be an advantage as they are much more simple to write and can be pushed to the upstream packages.

The initialization process will no longer be controlled by a bash script so it will be harder to skip a broken section. This is probably somewhat valid. I do not know enough about systemd to really comment beyond a couple of points. Firstly, systemd is (or will be) used by many distributions so will be less likely to break than our current initscripts. Also, systemd is quite modular, so it appears that stopping particular modules should be achievable.

It is written by Lennart Poettering. I have no idea how this ever became a “genuine” argument against systemd, but almost any discussion involving systemd ends up being about Lennart also writing pulseaudio. Maybe we need an updated version of Godwin’s Law for this situation… Any time an argument reaches this point, I know all things technical are out the door and I stop following.

When I was reading the old reviews, every time I saw mention of Arch Linux using the latest innovations in the Linux world, I started wondering why we are not using systemd already? Here I have only (partially) refuted the downsides of systemd, not pointed out what I can see are some quite substantial benefits. I personally think the historic rapid adoption of new features should continue in Arch Linux.

This has been a much longer post than I initially intended, but I hope this makes people think about what makes Arch Linux what it is. Or at least reconsider whether changes being made are really going against historical Arch principles. It has made me consider why I have not tried systemd yet, so I will do so in the next couple of weeks. Being on the bleeding edge is not much fun without the occasional cut…