Updating Arch Kernel On Digital Ocean

I have not bothered updating my Linux kernel package here since my VPS provider – Digital Ocean – went crazy and deprecated Arch Linux as an install option. That is obviously a bad thing…

This backfired on me today. It turns out that systemd-218 does not like an old i686 kernels (I had 3.13.7) and I could not log in. I am not the only one to notice this, so there may be a genuine bug there. It also seems that DO recovery console is a bit useless if you have a problem other than a network issue. Good thing I have DO system backups set to run every week (in a completely unrelated incident… I think one comment on one of my posts got lost and restoring from my more regular WordPress backup is too much effort).

So how to run a newer kernel – I have no idea what DO do in the background, but your installed kernel needs to match the one you select in the DO control panel. I think the kernel and initrd are stored externally and if this does not match the installed version, your system’s modules fail to load. This is how I know the rescue console is good when networking fails!

Kexec to the rescue! You can do a fancy systemd type thing (see Arch wiki page), but I just provided a script in /usr/bin/init. To do this, install kexec-tools and remove systemd-sysvcompat. Then paste this into /usr/bin/init:

kexec --load /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --append="root=LABEL=DOROOT init=/usr/lib/systemd/systemd" &&
mount -o ro,remount / &&
kexec -e
exec /usr/lib/systemd/systemd

Give that file 755 permissions, and all is done.

Now if all that does not put you off, and you really want to install Arch Linux on DO, here is a script that coverts a Debain droplet to an Arch one! You can also use my referral link

Edit: when checking this article before posting I found this and this. I’m not as clever as I thought, or these people are at least equally clever.

Who You Gonna Call?


Nobody… Arch Linux has the latest glibc release (2.20) so is not affected by the GHOST bug, which was fixed upstream in glibc-2.18 (without realising the security implications).

For those that want to know more about security issues and updates in Arch Linux, check out the arch-security mailing list.

Two PGP Keyrings for Package Management in Arch Linux

Both the pacman package manager and the makepkg tool for building packages verify files using PGP signatures. However, these two pieces of software do it using different keyrings. There seems to be a lot of confusion about this and misinformation is spreading at a rapid pace, so I’ll attempt to clarify it here!

Pacman Package File Signature Verification
By default, pacman is set-up to verify every package using a PGP signature. It has its own keychain for this purpose, located at /etc/pacman.d/gnupg/. This keychain is initialized during the Arch Linux install – a root key is created and the Arch Linux master keys are locally signed by the root key. The master keys sign all Arch Developer and Trusted User keys, creating an effective web-of-trust from your pacman root key to each of the packager keys allowing verification of package files.

If you want to allow the installation of package files from a non-official repository, you need to either disable signature verification (don’t do that…), or trust the packagers signing key. To do this you first need to verify their key ID, which should be well publicized. Then you import it into the pacman keyring using “pacman-key --recv-key <KEYID>” and signify that you trust the key by locally signing it with your pamcan root key by running “pacman-key --lsign <KEYID>“.

Makepkg Source File Signature Verification
When building a package, the source files are often (and should be!) signed, with a signature file available for download alongside the source file. This typically has the same name as the source file with the extension .sig or .asc.makepkg will automatically verify the signature if it is downloaded in the sources array. e.g.:


However, makepkg needs some information to verify the source signature. It will need the public PGP key of the person who signed the source file, and that key to be trusted. The difference here is that you do not trust whoever provided the source file to provide packages for your system (or at least you should not the vast majority of the time), so your user’s keyring is used. To get the key use “gpg --recv-key <KEYID>” and trust it (once suitably verified) using “gpg --lsign <KEYID>“.

If you provide a package to the AUR, it would be a lot of work for everyone to suitably verify a PGP key and locally sign it. To demonstrate that you have verified the key, you can add the following to the PKGBUILD:

validpgpkeys=('F37CDAB708E65EA183FD1AF625EF0A436C2A4AFF') # Carlos O'Donell

Now makepkg will trust that key, even if it is not trusted in the package builder’s PGP keyring. The builder will still need to download the key, but that can be automated in their gpg.conf file.

Hopefully that clarifies the two separate types of PGP signature verification happening in pacman and makepkg and explains why they should be separate… Now can people stop recommending that the pacman keyring is imported into the user’s keyring and vice versa?

Shellshock and Arch Linux

I’m guessing most people have heard about the security issue that was discovered in bash earlier in the week, which has been nicknamed Shellshock. Most of the details are covered elsewhere, so I thought I would post a little about the handling of the issue in Arch.

I am the Arch Linux contact on the restricted oss-securty mailing list. On Monday (at least in my timezone…), there was a message saying that a significant security issue in bash would be announced on Wednesday. I let the Arch bash maintainer and he got some details.

This bug was CVE-2014-6271. You can test if you are vulnerable by running

x="() { :; }; echo x" bash -c :

If your terminal prints “x“, then you are vulnerable. This is actually more simple to understand than it appears… First we define a function x() which just runs “:“, which does nothing. After the function is a semicolon, followed by a simple echo statement – this is a bit strange, but there is nothing stopping us from doing that. Then this whole function/echo bit is exported as an environmental variable to the bash shell call. When bash loads, it notices a function in the environment and evaluates it. But we have “hidden” the echo statement in that environmental variable and it gets evaluated too… Oops!

The announcement of CVE-2014-6271 was made at 2014-09-24 14:00 UTC. Two minutes and five seconds later, the fix was committed to the Arch Linux [testing] repository, where it was tested for a solid 25 minutes before releasing into our main repositories.

About seven hours later, it was noticed that the fix was incomplete. The simplified version of this breakage is

X='() { function a a>\' bash -c echo

This creates a file named “echo” in the directory where it was run. To track the incomplete fix, CVE-2014-7169 was assigned. A patch was posted by the upstream bash developer to fix this issue on the Wednesday, but not released on the bash ftp site for over 24 hours.

With a second issue discovered so quickly, it is important not to take an overly reactive approach to updating as you run the risk of introducing even worse issues (despite repeated bug reports and panic in the forum and IRC channels). While waiting for the dust to settle, there was another patch posted by Florian Weimer (from Red Hat). This is not a direct fix for any vulnerability (however see below), but rather a hardening patch that attempts to limit potential issues importing functions from the environment. During this time there was also patches posted that disabled importing functions from the environment all together, but this is probably an over reaction.

About 36 hours after the first bug fix, packages were released for Arch Linux that fixed the second CVE and included the hardening patch (which upstream appears to be adopting with minor changes). There were also two other more minor issues found during all of this that were fixed as well – CVE-2014-7186 and CVE-2014-7187.

And that is the end of the story… Even the mystery CVE-2014-6277 is apparently covered by the unofficial hardening patch that has been applied. You can test you bash install using the bashcheck script. On Arch Linux, make sure you have bash>=4.3.026-1 installed.

And because this has been a fun week, upgrade NSS too!

Who Packaged for Arch Linux in 2013

I was having a look at the current state of the Arch Linux repositories today in terms of the number of packages each person maintains and thought it interesting to see who did the packaging last year. So here are some numbers!

Firstly, the real repos (i.e. the repositories that TUs can not touch!). Note that the y-axis in this plot is the number of commits made to the repos and not the number of packages updated. Updating a package generally takes two commits and additional commits are done every time a package moves between repositories (e.g. moving packages out of [testing])

First, the two most prolific committers are Andrea Scarpino and Sven-Hendrik Haase. They both package KDE, which is in itself a lot of packages, but they get bonus commits for the [kde-unstable] repository where a lot of beta and release candidates are packaged. There is a lot of scripting going on for those rebuilds too, so don’t give them too much credit! Sven also deals with boost lately and the required rebuilds.

In places 3 and 6 are Jan Steffens and Jan de Groot who do our GNOME packaging. Rounding out the major desktop packagers is Evangelos Foutras in 9th place. In 4th, we have Andreas Radke who packages Xorg among other things including LibreOffice.

Eric Bélanger takes 5th place. I think he needs a specific shout out here because of all the effort he puts into maintaining the packages that officially have no maintainer. He regularly updates these packages and fixes their bugs. He also does far more than his share during rebuilds.

I am in 7th. This appears due to rebuilds for the removal /sbin et al., and the static libraries removal I pushed this year. In 8th is Tobias Powalowski who maintains the kernel package and deals with most of the module rebuilds.

Now a quick look at TU controlled repository commits. This includes the [community] and [multilib] repositories.

Not surprisingly, in first place we have Sergej Pupykin. He maintains about a 1/3 of the packages in the [community], although only has 1/6 of the commits… In 2nd place is Alexander Rødseth, who as far as I can tell does not maintain any specific package groups, so is just working hard! Bartłomiej Piotrowski is in 3rd (and who also rounded out the top 10 for the main repos) and we see Sven-Hendrik Haase again in 4th.

I’d also like to note the importance of all (or at least some!) of the people who have relatively few commits. In fact, I think we need more of them. I’d like to see the [extra] repo be almost exclusively the big package groups (Xorg, KDE, GNOME, XFCE, perl, python, etc) and [community] be all the additional packages with many more packagers each being responsible for a handful of packages that they are really interested in. So if you are thinking of applying for a Trusted User position, look at the tail of the distribution and do not let the big packagers put you off.

Shout-Out To Me on LAS (Not Really…)

For those that have not seen it, this week the Linux Action Show did a a wrap-up of their week long Arch challenge. If you are interested in that bit, start here… With praise such as “it is really not that bad”, I guess Arch Linux is doing well!

I got two indirect shout-outs. Clearly I am the developer who has an “ounce of contempt“. I live in the metric world, so that is 28.3495g, which seems relatively little. I also get a shout-out in their IRC screen! And to be clear, I am not a “my way is always right type”, it just works out that way.

Ways To Contribute To Arch Linux

I have been asked quite often recently “what is some ways I can contribute to Arch Linux?”. And the usual answers apply – package for the AUR, contribute to the wiki, help out on the forums and IRC… All useful to the whole community, but being not selfless at all I really want people to help in a way that helps me! So here are some ideas:

#1 – Fix Bugs on the Bug Tracker

This is not a new suggestion either… but it is an area that I think most people could contribute. I would say that there is some things that can be done to reduce the work for the developer that they are assigned to for the majority of bug reports in our tracker. For example:

  • Check that you can (still) replicate the bug. If you can, do not post “ME TOO!” as a comment. Just add a vote. Unless of course you can provide more information.
  • Summarize links to external reports. I hate bugs that are reported with just a link to a forum post or another bug tracker that requires me to read pages of posts to get at the issue. A brief summary of how to replicate is very useful.
  • For upstream bugs, make sure they are reported upstream. Provide the link to the upstream bug report or mailing list, etc.
  • If upstream has a patch for the issue, test it. If it works, provide the updated PKGBUILD and patch in the bug tracker, making sure a there is a comment in the PKGBUILD about where the patch came from.

#2 – Ensuring Continuously Buildable Arch

Arch rolls along a breakneck speed and quite often a package that was built a few weeks ago will no longer build from ABS. Then we get bug reports in the form “foo does not build”. In fact, most of those bug reports do not even provide build error (stupid users…)! These bugs particularly annoy me because I really dislike ABS in its current form. However, having packages that always build is very helpful when we do rebuilds due to library soname bumps or some change in packaging policy.

What I would propose is forming a group of people who in some way are continuously rebuilding Arch packages. That should be easy to script with the use of our devtools scripts. Any time a package no longer builds, the issue is investigated, reported upstream (if needed) and a fixed PKGBUILD is posted on the bug tracker. That way the developer only needs to quickly scan the updated PKGBUILD, confirm the fix is good, and push the rebuild to the repos.

#3 – Forming an Arch Security Team

Currently, Arch has a rather ad-hoc approach of dealing with security issues. This is because the majority of security fixes result in a new version of software being released, so we get them quickly with an update anyway. Also, developers tend to follow the mailing lists of the software they maintain. However, Arch could benefit from team who keep track of new security issues and make sure our packages get fixed. It would be worth doing a review of the current package situation too…

There was a project called “Arch Sheriff” a while ago that attempted to do this. However, I think it failed for two main reasons. It required the developers to go to an external website to check if their packages were vulnerable and, to a much lesser extent, it required them to go find the upstream fixes. A much better approach would be for the security team to interact with the developers through the bug tracker, providing fixed PKGBUILDs that include links to the issue and the upstream fix as comments.

#4 – Buy Allan Beer

On second thoughts, that probably hinders rather than helps…

For ideas #2 and #3, there will be some groundwork needed to get this underway. I guess using IRC, the wiki and mailing-lists should be enough for organizing how they will be run. I would not be directly involved in any of those ideas (they are suppose to free up my time!) but can advise and provide a middle man to the developers if needed. Continuously building packages for idea #2 would also require computing resources, but I guess that could be spread across people in the group.

So what are you waiting for? Get doing stuff!

Death of the [allanbrokeit] repository

I have deleted the [allanbroke] repository. It was started mainly to test the PGP signing implementation in pacman, which is now well established. Also, I would delay any packaging of release candidates or beta releases for this repository until I had enough free time and often official releases were made before that happened.

The repository may return someday, perhaps with VCS builds of packages I use locally once I get around to automating their creation as that would require no extra work…

My Arch Linux Talk at SINFO XX

I was recently invited to give a talk about Arch Linux at SINFO XX at IST in Lisbon, Portugal. It was a whirlwind tour of Europe, with the time I spent in transit almost exactly equal to the time I spent there.

Check out the video of my talk on their YouTube channel. I discuss what makes Arch different from other Linux distributions, what our strengths are as a distribution and briefly cover what future plans people have. I’m not going to watch it as it is never a good idea to dissect talks too much, so I’ll just assume I was awesome… It was also after midnight in my time zone, so I blame any mistakes on that.

Thanks to the organizers for inviting me over!

Edit: Quite a few people have asked for copies of me slides. Here they are (CC BY-SA): ODP PDF

Battle of the Arch Spin-Offs

According to Distrowatch, there are eleven “distributions” “based” on Arch Linux. I use “distributions” in quotes, because some are far less of a distribution than others and “based” gets quotes because some are so based that they are really Arch Linux in a poor disguise…

I have seen a bunch of new release announcements for several of these distributions in the last few days, so I thought I would take some for a spin and see what I am missing out on.

All distributions were tested in VirtualBox with a 128GB disk and 2GB RAM.

Contender #1: Chakra – 2012.12 Claire

The Chakra Project is my #1 contender as it is a real Linux distribution. They do use our PKGBUILDs (sometimes too directly…) but they rebuild everything for themselves. Chakra also has quite an interesting take on the rolling release model, opting for a “stable” core of the distribution and rolling release userland. So toolchain updates are rare, but the newest KDE will be packaged within days. I have issues with their “stable” core, because stable means no updates at all. Even minor bug fix updates are not considered and I am sure there is no backporting of any security fixes either as their team is too small to handle that. For example, I would not be running pacman-4.0.2 or even an unpatched pacman-4.0.3 as my distributions package manager. Another thing unique to Chakra is the pure KDE set-up. Any popular software that is not Qt based (e.g. firefox, GIMP) is supplied in a bundle. I find these a very bad idea, as many of the same libraries are on your system multiple times but in separate bundles, and it also appears that these are easy to break and under-maintained.

Onto the installer… Chakra wins points for being the only pure graphical installer out of the distributions I consider here. I did a review of the installer early in 2010 and there seemed a complete lack of improvement. The Live DVD booted fine and welcomed me with a window that pointed out the difference between Chakra and Arch, and noted that pacman is not designed for a GUI frontend (debatable…) so it will be replaced one day. Starting the install gives three screens worth of install notes which no-one has ever read and if anyone did read them, I am confident they did not care that their website is being redesigned. The partitioning took me to an external tool (KDE Partition Manager), which I actually found rather difficult to use. It took me several attempts to get everything set so that the installer would let me continue.

Once I finished the install, I rebooted… to a prompt. Whatever autodetection is done to get the KDE session up a running post-install had failed in VirtualBox. So no further review was done.

Contender #2: CinnArch – 2012.12.21

I had heard good things about the Cinnamon desktop, so CinnArch was my next choice. This distribution is Arch Linux – when I look at the package information for glibc, I see that I packaged it. The only difference is that an additional repo is provided with the packages needed for Cinnamon desktop to run.

The installer boots to a nice live system. There is two installation options – “CLI installer” and “Graphical Installer” but the latter is not selectable so “CLI” it is… In this case, CLI means an ascii-graphical menu driven installation environment that was easy to follow. The only steps that I found amusing was finding the fastest mirror to install from, which (with only slight exaggeration) took longer to complete than downloading the packages from the slowest mirror would have. And then on with installing the packages. Or not… It turns out that the installer detects I am using VirtualBox and tries to install the virtualbox-guest-modules package, which currently has a broken dependency and so can not be installed. There is no way to skip this that I could find.

So lets play around in the live media to see what might have been. It turns out that CinnArch provides two repos: cinnarch-repo and cinnarch-core. I could not find the justification for requiring a second repo. The repo looks fairly good on a quick glance, with all packages signed and a keyring provided. Theoretically, you can just enable that repo and install cinnamon from there, but I could not find where to see a list of keys that the developers use that would enable me to install the keyring package and then verify the rest of the packages. One thing I was concerned about was the filesystem package that was being provided for reasons I could not ascertain. The biggest issue with providing a repo for Arch Linux is that you need to keep up with our rolling base. That means your repo needs to have rebuilds available when we push (e.g.) a library update with an soname change. Providing an unnecessary filesystem package seems to only make things difficult for them. This was also my first time seeing the Pacman-XG GUI for the pacman package manager. That is enough said about that…

Contender #3: Manjaro – 0.8.3 XFCE edition

I was not expecting much from Manjaro given I had read this review of a previous installer. But given the “success” of the last two distros, it has a chance… Some things from that review had not changed – in the live environment there is still the unnecessary password of “manjaro” for both the user and root. And I was not endured by the green colour scheme or the logo which is used for the “start” menu. But to give Manjaro credit, I successfully installed using the old style Arch ascii-graphical install, which was easy for those that are use to that. Reiser3 as the default filesystem still confuses me.

So what do you get from Manjaro? It seems to be Arch delayed plus extras. The idea is that the Arch repos are delayed reaching you for a few days while everything is sorted out to give you an easy upgrade. This happens by adding a package manjaro-system which is always upgraded first and runs a script that automatically handles any manual intervention that would be required on a pure Arch Linux system. As a heads up, this uses a feature of pacman called SyncFirst that is removed for the upcoming pacman-4.1, so they may need to rethink their entire system soon.

The first difference I noted was it ran a 3.4.24 kernel, so that have held that back from being updated to the newest release, but at least it is the newest 3.4 series release. Looking at the glibc package once again, I am the packager, but it is a repo called [platform]. The Arch [extra] and [community] repos appear to be used as-is, so the need for [basis], [platform] and [addon] repos to replace our small [core] repo is strange. Also, this was the first time I had ever heard of a GUI for pacman with the imaginative name pacman-gui. It allows you to run pacman -Sy, which just refreshes the databases – a crazy thing to do without immediately updating in a rolling release distribution that does not provide multiple versions of a library. And it is not much of a GUI either, as it just launches a terminal that runs the pacman command you just clicked. There was also a LibreOffice installer that called pacman in the background.

The Winner?

Sadly, I do not think there was any winners today. Chakra had the most polished live environment, but failed to boot to the desktop. CinnArch failed through no fault of its own, which is sort of a fault of its own for not being a real distribution. Manjaro installed, although I saw nothing to make me want to recommend it.

The may be the most painful statement I have ever made… but ArchBang might be the winner!

Edit: Twelve hours later and the bug on the Arch end preventing the CinnArch install has been fixed. CinnArch installed and booted to the desktop without an issue.