Category Archives: Arch Linux

Pacman 6.0.0alpha1

We have been slowly working towards the next pacman release. The major change for this release is the implementation of parallel downloads. Check out this video:

It is time for this feature to receive some wider testing. Grab the package using:
pacman -U http://allanmcrae.com/packages/pacman-6.0.0alpha1-1-x86_64.pkg.tar.zst

This should be “safe” – there are quite a number of us using this code for our system management. There may be issues with AUR helpers due to ABI changes… If you are concerned, keep a copy of the repo package around to reinstall if necessary.

There are still a few issues with the downloader we will fix before release. If one of your mirrors is broken, you will be spammed with error messages at the end of the downloads. And TotalDownload does not work. But otherwise, it is good to go!

Report any bugs to the bug tracker.

The Case of GCC-5.1 and the Two C++ ABIs

Recently, Arch Linux updated to gcc-5.1. This brought a lot of new features, but the one that I am focusing on today is the new C++ ABI that appears when the compiler is built using default option.

Supporting the C++11 standard required incompatible changes to libstdc++ ABI. Instead of bumping the library soname (I still do not understand why that was not done…), the GCC developers decided to have a dual ABI. This is achieved using C++11 inlined namespaces which will give a different mangled name allowing types that differ across the two ABIs to coexist. When that is insufficient, a new abi_tag attribute is used which adds [abi:cxx11] to the mangled name.

With the initial upload of gcc-5.1 to the Arch repos, we configured GCC to use the old ABI by default. Now the initial slew of bugs have been dealt with (or mostly dealt with…), we want to switch to the new ABI. Because this was going to be a much larger rebuild than usual, one of the Arch developers (Evangelos Foutras) developed a server that automatically orders the rebuilds, and provides the next rebuild when a client requests it (this may be the future of rebuild automation in Arch).

This discovered an issue when building software using the new C++ ABI with clang, which builds against the new ABI (as instructed in the GCC header file), but does not know about the abi_tag attribute. This results in problems such as (for example) any function in a library with a std::string return type will be mangled with a [abi:cxx11] ABI tag. Clang does not handle these ABI tags, so will not add the tag to the mangled name and then linking will fail.

This issue was pointed out on a GCC mailing list in April, and a bug was filed independently for LLVM/clang. Until it is fixed, Arch Linux will not be able to switch to the new ABI. Note, this also has consequences for all older versions of GCC wanting to compile C++ software against system libraries…

Updating Arch Kernel On Digital Ocean

I have not bothered updating my Linux kernel package here since my VPS provider – Digital Ocean – went crazy and deprecated Arch Linux as an install option. That is obviously a bad thing…

This backfired on me today. It turns out that systemd-218 does not like an old i686 kernels (I had 3.13.7) and I could not log in. I am not the only one to notice this, so there may be a genuine bug there. It also seems that DO recovery console is a bit useless if you have a problem other than a network issue. Good thing I have DO system backups set to run every week (in a completely unrelated incident… I think one comment on one of my posts got lost and restoring from my more regular WordPress backup is too much effort).

So how to run a newer kernel – I have no idea what DO do in the background, but your installed kernel needs to match the one you select in the DO control panel. I think the kernel and initrd are stored externally and if this does not match the installed version, your system’s modules fail to load. This is how I know the rescue console is good when networking fails!

Kexec to the rescue! You can do a fancy systemd type thing (see Arch wiki page), but I just provided a script in /usr/bin/init. To do this, install kexec-tools and remove systemd-sysvcompat. Then paste this into /usr/bin/init:

#!/bin/sh
 
kexec --load /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --append="root=LABEL=DOROOT init=/usr/lib/systemd/systemd" &&
mount -o ro,remount / &&
kexec -e
 
exec /usr/lib/systemd/systemd
Give that file 755 permissions, and all is done.

Now if all that does not put you off, and you really want to install Arch Linux on DO, here is a script that coverts a Debain droplet to an Arch one! You can also use my referral link…

Edit: when checking this article before posting I found this and this. I’m not as clever as I thought, or these people are at least equally clever.

Who You Gonna Call?

GHOSTbusters

Nobody… Arch Linux has the latest glibc release (2.20) so is not affected by the GHOST bug, which was fixed upstream in glibc-2.18 (without realising the security implications).

For those that want to know more about security issues and updates in Arch Linux, check out the arch-security mailing list.

Two PGP Keyrings for Package Management in Arch Linux

Both the pacman package manager and the makepkg tool for building packages verify files using PGP signatures. However, these two pieces of software do it using different keyrings. There seems to be a lot of confusion about this and misinformation is spreading at a rapid pace, so I’ll attempt to clarify it here!

Pacman Package File Signature Verification
By default, pacman is set-up to verify every package using a PGP signature. It has its own keychain for this purpose, located at /etc/pacman.d/gnupg/. This keychain is initialized during the Arch Linux install – a root key is created and the Arch Linux master keys are locally signed by the root key. The master keys sign all Arch Developer and Trusted User keys, creating an effective web-of-trust from your pacman root key to each of the packager keys allowing verification of package files.

If you want to allow the installation of package files from a non-official repository, you need to either disable signature verification (don’t do that…), or trust the packagers signing key. To do this you first need to verify their key ID, which should be well publicized. Then you import it into the pacman keyring using “pacman-key --recv-key <KEYID>” and signify that you trust the key by locally signing it with your pamcan root key by running “pacman-key --lsign <KEYID>“.

Makepkg Source File Signature Verification
When building a package, the source files are often (and should be!) signed, with a signature file available for download alongside the source file. This typically has the same name as the source file with the extension .sig or .asc.makepkg will automatically verify the signature if it is downloaded in the sources array. e.g.:

source=(http://ftp.gnu.org/gnu/libc/${pkgname}-${pkgver}.tar.xz{,.sig})

However, makepkg needs some information to verify the source signature. It will need the public PGP key of the person who signed the source file, and that key to be trusted. The difference here is that you do not trust whoever provided the source file to provide packages for your system (or at least you should not the vast majority of the time), so your user’s keyring is used. To get the key use “gpg --recv-key <KEYID>” and trust it (once suitably verified) using “gpg --lsign <KEYID>“.

If you provide a package to the AUR, it would be a lot of work for everyone to suitably verify a PGP key and locally sign it. To demonstrate that you have verified the key, you can add the following to the PKGBUILD:

validpgpkeys=('F37CDAB708E65EA183FD1AF625EF0A436C2A4AFF') # Carlos O'Donell

Now makepkg will trust that key, even if it is not trusted in the package builder’s PGP keyring. The builder will still need to download the key, but that can be automated in their gpg.conf file.

Hopefully that clarifies the two separate types of PGP signature verification happening in pacman and makepkg and explains why they should be separate… Now can people stop recommending that the pacman keyring is imported into the user’s keyring and vice versa?

Shellshock and Arch Linux

I’m guessing most people have heard about the security issue that was discovered in bash earlier in the week, which has been nicknamed Shellshock. Most of the details are covered elsewhere, so I thought I would post a little about the handling of the issue in Arch.

I am the Arch Linux contact on the restricted oss-securty mailing list. On Monday (at least in my timezone…), there was a message saying that a significant security issue in bash would be announced on Wednesday. I let the Arch bash maintainer and he got some details.

This bug was CVE-2014-6271. You can test if you are vulnerable by running

x="() { :; }; echo x" bash -c :

If your terminal prints “x“, then you are vulnerable. This is actually more simple to understand than it appears… First we define a function x() which just runs “:“, which does nothing. After the function is a semicolon, followed by a simple echo statement – this is a bit strange, but there is nothing stopping us from doing that. Then this whole function/echo bit is exported as an environmental variable to the bash shell call. When bash loads, it notices a function in the environment and evaluates it. But we have “hidden” the echo statement in that environmental variable and it gets evaluated too… Oops!

The announcement of CVE-2014-6271 was made at 2014-09-24 14:00 UTC. Two minutes and five seconds later, the fix was committed to the Arch Linux [testing] repository, where it was tested for a solid 25 minutes before releasing into our main repositories.

About seven hours later, it was noticed that the fix was incomplete. The simplified version of this breakage is

X='() { function a a>\' bash -c echo

This creates a file named “echo” in the directory where it was run. To track the incomplete fix, CVE-2014-7169 was assigned. A patch was posted by the upstream bash developer to fix this issue on the Wednesday, but not released on the bash ftp site for over 24 hours.

With a second issue discovered so quickly, it is important not to take an overly reactive approach to updating as you run the risk of introducing even worse issues (despite repeated bug reports and panic in the forum and IRC channels). While waiting for the dust to settle, there was another patch posted by Florian Weimer (from Red Hat). This is not a direct fix for any vulnerability (however see below), but rather a hardening patch that attempts to limit potential issues importing functions from the environment. During this time there was also patches posted that disabled importing functions from the environment all together, but this is probably an over reaction.

About 36 hours after the first bug fix, packages were released for Arch Linux that fixed the second CVE and included the hardening patch (which upstream appears to be adopting with minor changes). There were also two other more minor issues found during all of this that were fixed as well – CVE-2014-7186 and CVE-2014-7187.

And that is the end of the story… Even the mystery CVE-2014-6277 is apparently covered by the unofficial hardening patch that has been applied. You can test you bash install using the bashcheck script. On Arch Linux, make sure you have bash>=4.3.026-1 installed.

And because this has been a fun week, upgrade NSS too!

Who Packaged for Arch Linux in 2013

I was having a look at the current state of the Arch Linux repositories today in terms of the number of packages each person maintains and thought it interesting to see who did the packaging last year. So here are some numbers!

Firstly, the real repos (i.e. the repositories that TUs can not touch!). Note that the y-axis in this plot is the number of commits made to the repos and not the number of packages updated. Updating a package generally takes two commits and additional commits are done every time a package moves between repositories (e.g. moving packages out of [testing])

First, the two most prolific committers are Andrea Scarpino and Sven-Hendrik Haase. They both package KDE, which is in itself a lot of packages, but they get bonus commits for the [kde-unstable] repository where a lot of beta and release candidates are packaged. There is a lot of scripting going on for those rebuilds too, so don’t give them too much credit! Sven also deals with boost lately and the required rebuilds.

In places 3 and 6 are Jan Steffens and Jan de Groot who do our GNOME packaging. Rounding out the major desktop packagers is Evangelos Foutras in 9th place. In 4th, we have Andreas Radke who packages Xorg among other things including LibreOffice.

Eric Bélanger takes 5th place. I think he needs a specific shout out here because of all the effort he puts into maintaining the packages that officially have no maintainer. He regularly updates these packages and fixes their bugs. He also does far more than his share during rebuilds.

I am in 7th. This appears due to rebuilds for the removal /sbin et al., and the static libraries removal I pushed this year. In 8th is Tobias Powalowski who maintains the kernel package and deals with most of the module rebuilds.

Now a quick look at TU controlled repository commits. This includes the [community] and [multilib] repositories.

Not surprisingly, in first place we have Sergej Pupykin. He maintains about a 1/3 of the packages in the [community], although only has 1/6 of the commits… In 2nd place is Alexander Rødseth, who as far as I can tell does not maintain any specific package groups, so is just working hard! Bartłomiej Piotrowski is in 3rd (and who also rounded out the top 10 for the main repos) and we see Sven-Hendrik Haase again in 4th.

I’d also like to note the importance of all (or at least some!) of the people who have relatively few commits. In fact, I think we need more of them. I’d like to see the [extra] repo be almost exclusively the big package groups (Xorg, KDE, GNOME, XFCE, perl, python, etc) and [community] be all the additional packages with many more packagers each being responsible for a handful of packages that they are really interested in. So if you are thinking of applying for a Trusted User position, look at the tail of the distribution and do not let the big packagers put you off.

Shout-Out To Me on LAS (Not Really…)

For those that have not seen it, this week the Linux Action Show did a a wrap-up of their week long Arch challenge. If you are interested in that bit, start here… With praise such as “it is really not that bad”, I guess Arch Linux is doing well!

I got two indirect shout-outs. Clearly I am the developer who has an “ounce of contempt“. I live in the metric world, so that is 28.3495g, which seems relatively little. I also get a shout-out in their IRC screen! And to be clear, I am not a “my way is always right type”, it just works out that way.

Ways To Contribute To Arch Linux

I have been asked quite often recently “what is some ways I can contribute to Arch Linux?”. And the usual answers apply – package for the AUR, contribute to the wiki, help out on the forums and IRC… All useful to the whole community, but being not selfless at all I really want people to help in a way that helps me! So here are some ideas:

#1 – Fix Bugs on the Bug Tracker

This is not a new suggestion either… but it is an area that I think most people could contribute. I would say that there is some things that can be done to reduce the work for the developer that they are assigned to for the majority of bug reports in our tracker. For example:

  • Check that you can (still) replicate the bug. If you can, do not post “ME TOO!” as a comment. Just add a vote. Unless of course you can provide more information.
  • Summarize links to external reports. I hate bugs that are reported with just a link to a forum post or another bug tracker that requires me to read pages of posts to get at the issue. A brief summary of how to replicate is very useful.
  • For upstream bugs, make sure they are reported upstream. Provide the link to the upstream bug report or mailing list, etc.
  • If upstream has a patch for the issue, test it. If it works, provide the updated PKGBUILD and patch in the bug tracker, making sure a there is a comment in the PKGBUILD about where the patch came from.

#2 – Ensuring Continuously Buildable Arch

Arch rolls along a breakneck speed and quite often a package that was built a few weeks ago will no longer build from ABS. Then we get bug reports in the form “foo does not build”. In fact, most of those bug reports do not even provide build error (stupid users…)! These bugs particularly annoy me because I really dislike ABS in its current form. However, having packages that always build is very helpful when we do rebuilds due to library soname bumps or some change in packaging policy.

What I would propose is forming a group of people who in some way are continuously rebuilding Arch packages. That should be easy to script with the use of our devtools scripts. Any time a package no longer builds, the issue is investigated, reported upstream (if needed) and a fixed PKGBUILD is posted on the bug tracker. That way the developer only needs to quickly scan the updated PKGBUILD, confirm the fix is good, and push the rebuild to the repos.

#3 – Forming an Arch Security Team

Currently, Arch has a rather ad-hoc approach of dealing with security issues. This is because the majority of security fixes result in a new version of software being released, so we get them quickly with an update anyway. Also, developers tend to follow the mailing lists of the software they maintain. However, Arch could benefit from team who keep track of new security issues and make sure our packages get fixed. It would be worth doing a review of the current package situation too…

There was a project called “Arch Sheriff” a while ago that attempted to do this. However, I think it failed for two main reasons. It required the developers to go to an external website to check if their packages were vulnerable and, to a much lesser extent, it required them to go find the upstream fixes. A much better approach would be for the security team to interact with the developers through the bug tracker, providing fixed PKGBUILDs that include links to the issue and the upstream fix as comments.

#4 – Buy Allan Beer

On second thoughts, that probably hinders rather than helps…

For ideas #2 and #3, there will be some groundwork needed to get this underway. I guess using IRC, the wiki and mailing-lists should be enough for organizing how they will be run. I would not be directly involved in any of those ideas (they are suppose to free up my time!) but can advise and provide a middle man to the developers if needed. Continuously building packages for idea #2 would also require computing resources, but I guess that could be spread across people in the group.

So what are you waiting for? Get doing stuff!