Nothing like a new pacman release to make me locate the password to this site…

Tradition dictates I thank people who have contributed to the release (as well as genuinely meaning the thanks!). We had 29 people have a patch committed this release, with a few new names. Here is the top ten:

$ git shortlog -n -s v5.1.0..v5.2.0 | head -n10
   108  Eli Schwartz
    38  Allan McRae
    30  morganamilo
    24  Andrew Gregory
    20  Dave Reisner
     9  Jan Steffens
     6  Michael Straube
     4  Jonas Witschel
     4  Luke Shumaker
     3  Que Quotion


We have a clear winner. Although I’m sure that at least half of those are in responses to bugs he created! He claims it is a much smaller proportion… And a new contributor in third.

What has changed in this release? Nothing super exciting as far as I’m concerned, but check out the detailed list here.

We have completely removed support for delta packages. This was a massively underused feature, usually made updates slower for a slight saving on bandwidth, and had a massive security hole. Essentially, a malicious package database in combination with delta packages could run arbitrary commands on your system. This would be less of an issue if a certain Linux distro signed their package databases… Anyway, on balance I judged it better to remove this feature altogether. We may come back to this in the future with a different implementation, but I would not expect that any time soon. Note a similar vulnerability was found with using XferCommand to download packages, but we plugged that hole instead of removing it!

Support for downloading PGP keys using the new Web Key Directory (WKD) was added to pacman. Both pacman-key and makepkg will also look there by default with the latest GnuPG release. This prevents DoS attacks through people adding very large numbers of signatures to PGP keys. The attack scope was limited for Arch Linux anyway, as most people obtain the pacman keyring through the archlinux-keyring package.

The much maligned --force made its way to /dev/null. The --overwrite option has been a replacement for over a year and is a precision surgical instrument compared to the blunt hammer of --force.

There is a small user interface change for searching files databases with -F. Specifying the -s option was redundant, so removed. More information such as package group and installed status is shown in the search results, bringing the output inline with -Ss.

The split of makepkg into smaller and extendable components continued. You can now provide new source download and signature verification routines (e.g. if you are living in the past and want to support cvs:// style URLs). We also added support for lzip, lz4 and zst compressed packages. Arch Linux will switch zst by default in the near future.

Under the hood, we are in the process of changing our build system from autotools to meson. This is relatively complete, but there still was a decent churn of patches to meson files as we approached release. You can build pacman from the release tarball using meson if you want to test. Next release is likely to be meson only. (Edit: you can’t test meson with the 5.2.0 tarball as it is missing a couple of the meson build files.)

Expect the release to land in Arch Linux “soon”. Expect to see another blog post in a year or so when I make the next release…

Wow… look at all the cobwebs around here! No posts in two years. But the need for a pacman release post has dragged me back. I clearly still remembered the password, so that is a bonus!

As is tradition, before I get in to details, I need to thank everyone for their help in making this release. Here are the top 10 committers:

$ git shortlog -n -s v5.0.0..v5.1.0
    82  Allan McRae
    60  Andrew Gregory
    45  Eli Schwartz
    16  Ivy Foster
    10  Dave Reisner
     9  Christian Hesse
     9  Gordian Edenhofer
     8  Alastair Hughes
     7  Rikard Falkeborn
     6  Michael Straube

(I win!) Lots of new names there which is always really appreciated. And as usual a long tail of contributors submitting the occasional patch – there were 48 contributors in total.

Onto what has changed in this release. There is a lack of what I would call a killer feature in this release. Mostly a lot of small changes that improve usability, which is why there was so much time between releases. Here is a detailed list of changes. However, there are a few things worth highlighting.

There is a new option --overwrite, which is a replacement for to often misused --force (hence the release name). This allows fine grained control of what files pacman is safe to ignore conflicts with. Handling the latest upgrade requiring user intervention in Arch Linux would now look like:
pacman -Syu --overwrite usr/lib/libmozjs-52.so.0You can even use globs when specifying the files to overwrite. Not only is specifying exact files to overwrite a lot safer than the old --force, there are also some common sense restrictions there too (you can’t overwrite a directory with a file, or force package installs with conflicting files).

We have also added a --sysroot option that will replace --root. Basically, this now works the way people will expect – for example, the configuration file used is the one in the specified root, and not the local one. This does require a bit more setup while creating a new install root, but hopefully will be a lot more robust.

We have also added support for reproducible builds. This was mostly ensuring all files had the same timestamp and obeyed the SOURCE_DATE_EPOCH standard. We also added a .BUILDINFO file within each package, recording information about the environment a package was built in. This allows scripts to regenerate the build environment to demonstrate a package is reproducible (particularly important in rolling release distros).

There was also improved support for debugging packages. Split packages now produce a single debug package instead of one for each split package. This makes it easier to get all required debug symbols for a particular package (and hopefully easier for distros to carry these packages…). Also, we include relevant source files in the debug packages, allowing us to step through the code.

Finally, I killed off the “contrib” directory as it was taking excessive amounts of pacman developer time. That means no more checkupdates, paccache, … However, this has been picked up as a separate project, which is available by installing pacman-contrib in Arch Linux.

As always, this is a bug free release. But if you spot something you think is a bug, please file a bug report and we can assign blame – which is more important than fixing! (The pool for developer who created the first pacman bug of this release is still open at the time of posting.)

As is becoming tradition, I need to make a blog post to accompany a pacman release! This is a big release with a long awaited feature so it needed a major version bump (and, most importantly, we now are back ahead of the Linux kernel in version numbers). I have reclaimed the title as most prolific committer, but that just means Andrew had more patches to point out mistakes in… Here are the top 10 committers:

$ git shortlog -n -s v4.2.0..v5.0.0
   176  Allan McRae
    85  Andrew Gregory
    16  Florian Pritz
     9  Dave Reisner
     9  Johannes Löthberg
     9  Rikard Falkeborn
     7  Pierre Neidhardt
     5  David Macek
     5  Evangelos Foutras
     4  Mohammad Alsaleh

As always, more regular contributors would be helpful. Just have a talk to us first before running ahead implementing a new feature (it is not nice to have to reject a patchset that obviously took a lot of work because it is already being handled in a different way…).

On to the more important stuff… What is new this release?

Hooks: This has been one of the most requested features for a long time, and Andrew gets all the credit for the implementation. So, what exactly are hooks? Hooks are scripts that are run at the beginning and end of a transaction. They can be triggered by either a file or a package name. This will allow us to (e.g.) update the desktop MIME type cache at the end of a transaction (if needed) and only do it one rather than after every package. Andrew has a git repo with some examples. Lets look at the desktop MIME cache one:

[Trigger]
Type = File
Operation = Install
Operation = Upgrade
Operation = Remove
Target = usr/share/applications/*.desktop
 
[Action]
When = PostTransaction
Exec = /bin/update-desktop-database --quiet

It should be fairly obvious what that does… See the alpm-hooks(5) man page for more information on the hook format.

Files database operations: pacman can now search repository file lists like pkgfile, but slower and probably less flexible. Sort of like Falcon to Captain America… still a super-hero! File bug reports for improvement requests. A separate files database is used so everyday package operations do not require downloading a much larger database. After updating the database (with “pacman -Fy“, you can do things like:

$ pacman -Fo /usr/bin/pacman
usr/bin/pacman is owned by core/pacman 4.2.1-4
 
$ pacman -Fl pacman
pacman etc/
pacman etc/makepkg.conf
pacman etc/pacman.conf
...
 
$ pacman -Fs libalpm.so
core/pacman 4.2.1-4
    usr/lib/libalpm.so
 
$ pacman -Fsx kcm.*print.*\.so
extra/print-manager 15.12.1-1
    usr/lib/qt/plugins/kcm_printer_manager.so
community/kmymoney 4.7.2-3
    usr/lib/kde4/kcm_kmm_printcheck.so

libmakepkg: makepkg in the pacman-4.2 release was a 3838 line shell script. A bit daunting, hard to test and not reusable… I have started the process of splitting this into a library containing scripts that are a more reasonable size. makepkg is still 2395 lines, so a lot of work remains (help!). One outcome of this splitting is we can drop in extra checks into the PKGBUILD and package checking steps, and even extra passes to (e.g.) optimize svg files. I have started rewriting namcap using this feature (see my github repo). This also provides tools for extracting variables from PKGBUILDs without sourcing the PKGBUILD itself, which does require ensuring that variables that should be arrays are actually arrays and those that are not are not.

There were a bunch of other small changes throughout the code base. Check out the NEWS file for more details.

An alarming number of people have noticed, the pacman-4.2 release removed the --asroot option from makepkg. This means that you can no longer build packages as the root user. There are good reasons for this and the option was only included due to issue we had building under fakeroot (only the package() function gets fun under fakeroot these days, and there has been no issues with fakeroot in a while anyway).

Even if your PKGBUILD file is not malicious, there are good examples of when something goes wrong by accident. Remember the bumblebee bug that deleted /usr due to an extra space? Or just this week a steam bug that deletes a user home directory? Do you still want to run code as root? OK then… I am going to show you how not to!

Firstly, we need a build directory. I suggest /home/build. Putting this directory directly under /root will not work unless you want to relax its 700 permissions to allow the nobody user read/write access¹. I suppose you could as you are running as root… but I will use /home/build. Create the directory and set permissions with the following:

mkdir /home/build
chgrp nobody /home/build
chmod g+ws /home/build
setfacl -m u::rwx,g::rwx /home/build
setfacl -d --set u::rwx,g::rwx,o::- /home/build


Not that people running makepkg as root need to know what code is doing to run it… I’ll explain what is happening here. Firstly create a /home/build directory, make it owned by the nobody group and ensure that group has write permissions. Also add the sticky flag to the group permissions so all files created in that directory also are owned by the nobody group. Then we set ACLs to ensure all files and directories created in /home/build have group read/write permissions.

Now to building you package! Get you PKGBUILD in your new build directory and run makepkg as the nobody user. You can do this using su but using sudo has the advantage of being able to alias this command. Installing sudo does not create a security risk as you are running as root! You also do not need to configure anything as root will have full sudo permissions by default². Build your package using:

sudo -u nobody makepkg

Done… I’d add “alias makepkg='sudo -u nobody makepkg” to your ~/.bashrc so you never have to type this again.

There is still a problem here. If you download and manually extract a package sourceball, or use an AUR helper such as cower to do so, the group write permissions get lost:

[root@arya build]# cower -d pacman-git
:: pacman-git downloaded to /home/build
 
[root@arya build]# ls -ld pacman-git/
drwxr-xr-x+ 2 root nobody 4096 Mar 21 2013 pacman-git/


Doing “chmod -R g+w pacman-git/” will fix this. There is probably a way to avoid this – at least when manually extracting the tarball, but I have no interest in figuring it out. Otherwise, it is a two line function.

And if this does not satisfy you, revert that patch that removed --asroot. It should still revert cleanly.

¹ makepkg checks directory write permissions using the full path so fails if any parent directories are not writable. I guess this could be fixed if someone was interested.

² Note that to have makepkg install missing dependencies and install your built package without being queried the password for the nobody user (which would be difficult to answer…), you will need to configure nobody to run sudo pacman without a password.

I released pacman-4.2 on the 19th of December – which is only marginally after the end of August as originally planned… We had 52 contributors provide patches to this release. Andrew takes the prize for most commits. Here are the top 10:

$ git shortlog -a -s -n v4.1.0..v4.2.0
   164  Andrew Gregory
   139  Allan McRae
    66  Dave Reisner
    26  Jason St. John
    20  Florian Pritz
    18  Pierre Neidhardt
    15  Olivier Brunel
     9  Jeremy Heiner
     9  Jonathan Frazier
     8  Dan McGee


The real prize goes to the person who caused the first reported bug. That could have been Dave but he caught it just in time. And I mean just! I posted to IRC “any ideas for the tag message” and the response I got was “I think I broke updpkgsums“. The shame of being first is inversely proportional to your commit count. (The small typos discovered so far do not count…)

Packaging Changes
There has been a couple of useful features added to makepkg. The main ones are:

Architecture Specific Fields: The source and depends (and related fields) now can all specify architecture specific values. For example:

source=("http://example.com/foo-$pkgver.tar.gz")
source_i686+=("http://example.com/bar32-$pkgver.tar.gz")
source_x86_64+=("http://example.com/bar64-$pkgver.tar.gz")

The source for a given architecture is used in addition to the global source. The ‘+=‘ when specifying extra sources for an architecture does nothing different than just using ‘=‘, but I use it to serve as a reminder that these are additional values. Thanks to Dave!

Templating PKGBUILDs: Many PKGBUILDs share a similar build system, making them highly redundant. This is an attempt to reduce the redundancy by providing a template system. The easiest way to describe this is using an example, so I will use a potential perl module template. We create a file /usr/share/makepkg-template/perl-module-1.0.template. In this file is the build(), check() and package() functions and any common biolerplate. As this is our current version, it is also symlinked to perl-module.template. In our PKGBUILD, we would add:

# template input; name=perl-module;

and run makepkg-template. Now look in the PKGBUILD and you will see that line is replaced with:

# template start; name=perl-module; version=1.0;
build() {
...
# template end;

If we ever need to update the template, we create perl-module-2.0.template and update the symlink. Now run makepkg-template -n to update the PKGBUILD. Read “man makepkg-template” for more details. Thanks to Florian!

Incremental VCS Builds: Previously makepkg would remove its working copy of the VCS source directory before starting a new build. Now makepkg will just update the source copy (or attempt to in the case of SVN…) and build the package. This brings VCS builds in line with those using non-VCS sources. A new option -C/--clean was added to makepkg to remove the old $srcdir before building for cases where incremental builds fail. Thanks to Lukáš (and sorry it took me so long to deal with your patches)!

Source Package Information: To avoid things like the AUR attempting to parse bash to display information from a source tarball, we now provide a .SRCINFO file in an easily parseable format. Thanks to Dave!

Package Functions are Mandatory : The use of package() functions in PKGBUILD was introduced a long time ago. Now it is mandatory that a PKGBUILD has one (with the exception being metapackages that do not have a build() function either). Now that fakeroot usage is limited to the packaging step, the use of fakeroot is mandatory and building as root is disabled.

Misc. Changes: Other things of interest:

• Static libraries are only removed with options=('!static') if they have a shared counterpart
• Source signatures are required to be from a trusted source or listed in the validpgpkeys array. We also support kernel.org style source signing
• Split packages can no longer override pkgver/pkgrel/epoch as that was a silly idea…

Pacman Changes

No we don’t have hooks… They are strongly planned for the next release.

Directory Symlink Handling: Example time! Arch Linux has a /lib -> /usr/lib symlink. Previously, if pacman was installing a package and it found files in /lib, it would follow the symlink and install it in /usr/lib. However the filelist for that package still recorded the file in /lib. This caused heaps of difficulty in conflict resolving – primarily the need to resolve every path of all package files to look for conflicts. That was a stupid idea! So now if pacman sees a /lib directory in a package, it will detect a conflict with the symlink on the filesystem. If you were using this feature to install files elsewhere, you probably need to look into what a bind mount is! Note that this change requires us to correct the local package file list for any package installed using this mis-feature, so we bumped the database version. Upgrade using pacman-db-upgrade. Thanks to Andrew!

Added an –assume-installed Option: I believe this options was invented during a perl update. Almost all compiled perl modules have a dependency on a specific perl version. So with a major perl update, all the modules need to be updated at the same time, or you can use -d to ignore dependency versions, but for all packages and not just perl. This is not a problem with the Arch repositories where all packages are updated at the same time, but if you have lots of perl modules from the AUR, you will need to remove those, update, then rebuild them. Instead you can use --assume-installed perl-5.18 and all those packages depending on perl=5.18 will not complain. Thanks to Florian!

Repository Usage Configuration: A new configuration keyword was added for repositories – Usage. It can take values Sync, Search, Install, Upgrade, All. For example, I have the [staging] and [multilb-testing] repositories in my pacman.conf with the Sync usage. That way I can look at what is in these repositories without using them for package updates. Thanks to Dave!

Mics. Changes: Other changes to pacman:

• Improved dependency ordering – the dependency ordering did not go deep enough into the tree to ensure correct installation order.
• A warning is printed if a directory on the filesystem has different permissions to the one being “installed” from the package.
• Lock files should now never be left behind…
• Various speed-ups, memory leak plugs and bug fixes

See here for a more complete list of changes.

And I have just realized that the only major change I contributed was the requiring of package() functions, which I am told means 1/3 of the AUR will not build! It feels good to be back to breaking things…

I was listening to Frostcast in the background today when I heard my name. That always makes me pay some attention. Then I heard wrong information. I don’t know why I care, but I do… so here goes the clarification.

The quote from Philip Müller at 14:35 into the podcast:

The lastest news was Allan McRae – he is a developer of pacman himself – he sent me an email to send over all the translations of Manjaro distribution does. So I forked pacman, and pacman itself has 20 translations and our branch has 44 translations of the same software so Arch Linux is asking us to be upstream and give them our translations…

OK… This is interesting. Time for some background here. When pacman-4.1 was released, we removed the broken SyncFirst option. This is needed by Manjaro Linux to run their update helper script that “fixes” the update process to remove any manual interventions. So Manjaro reverted our patch and brought back SyncFirst to pacman. That required three additional strings to be translated for their version of pacman so they also forked our translation project on Transifex.

As the Arch and Manjaro versions of these projects had started to diverge, I wrote to Phil noting that people were doing more than just translating those three additional strings, and it would be good if the translators were pointed at the Arch project so we all benefited, given the Arch project is the one the pacman developers set up.

Lets compare the status of the Arch and Manjaro translations as of 2013-09-24. There are 24 languages with complete translations in the Arch projects, and being nice and ignoring the additional three strings in the Manjaro project, they have 23. (Of those 23, only 6 actually have the additional three Manjaro strings translated). What are the differences? Manjaro has a complete Hungarian translation while Arch has complete Korean and Romanian translations. The Arch Hungarian translation is at 99%, while the Manjaro Korean and Romanian are at 21% and 62% respectively. So it is clear these languages have diverged since the split, with most of the work done in Arch.

Of the remaining languages with incomplete translations, Manjaro has 19 languages, while Arch has 15. Clearly not a total difference of 20 to 44 languages as claimed. Looking at these in more detail, 9 languages have not deviated between the two projects. The Arabic, Chinese (Taiwan), Dutch, Galician, Polish, Serbian (Latin) translations have all got additional translations in the Arch project since the split with the Manjaro project. So apart from languages that have been have had translations started in Manjaro but not in Arch, the Arch project is behind in 3 strings for the Hungarian language.

Maybe where the Arch translation project for pacman could gain is from the new languages in the Manjaro translation: Czech (Czech Republic) [99%], Bulgarian (Bulgaria) [62%], Uzbek [14%] and Danish (Denmark) [3%]. Also note that 3/4 of those languages have a sub-name there. Taking “Danish (Denmark)” as an example, there is already a “Danish” translation (language code: da) and this is adding a Denmark specialization (language code: da_DK). I might be entirely wrong here, but are there other variants of Czech, Bulgarian and Danish apart from their primary usage, or are these exactly the same and the work is just being repeated?

In summary, the translation project set up by the pacman developers is, and will remain, the upstream translation. I just approached Manjaro to send their translations our way so we would both benefit. Arch from (potentially) more translations, and it would be easier for Manjaro to merge their string translations without ending up removing several hundred perfectly good translations.

I will clarify this just because I have had several people ask me already. No, we did not remove the SyncFirst option in pacman to deliberately cause issues for Manjaro Linux. In fact, it was first discussed in Feburary 2012 and, as far as I can tell, Manjaro has only been around from late March 2012 (looking at the earliest commits in their git repository).

So lets keep the conspiracy theories to a minimum! (or at least come up with a better one…)

I have just released pacman-4.1 and packages are now in the [testing] repo. This is the first time I have made a release for any software project, so I was glad to have released a 4.1RC a few weeks back to learn everything that needed to be done.

It has been over a year since the pacman-4.0 release and there have been a large number of contributions made:

$ git shortlog -n -s --no-merges v4.0.0..v4.1.0
   239 Allan McRae
   185 Dan McGee
   158 Dave Reisner
    52 Andrew Gregory
    23 Simon Gomizelj
    20 William Giokas
    19 Florian Pritz
    15 Daniel Wallace
    ...


I win this time! Apart from the usual three contributors, it was great to see other people regularly helping out, both in providing and reviewing patches. A particular thanks to Andrew Gregory who helped me figure out how to fix something on several occasions and has been actively commenting on patches sent to the mailing list. His patch count also puts him in the top ten contributors of all time. In total we have 45 people with patches accepted for this release. Also a big thank you to our translators – particularly because I was learning how the system worked and may have required additional strings to be translated on a couple of occasions…

Moving on to what has changed. There have been quite a number of features added to pacman and makepkg and a couple of new helper scripts in this release.

The major feature for the release is tight integration between the package manager and systemd. After much discussion about how best to perform updates on a rolling release system, we realized that it was essential to have updates preformed with minimal other processes running. Also, the security aspects of updates mean that it is essential that these get provided as soon as possible. We felt the best way to achieve this was to perform updates on shutdown. This is achieved through a new daemon, pacmand that monitors and downloads updates in the background. When updates are found, it schedules a reboot of the system (hence the need to integrate systemd). At the moment the timing of the reboots is not configurable, but a timer will pop-up to allow you to delay it for a preset amount of time. Configuration will likely be added in pacman-4.2, when pacmanctl will be ready for general use. Until that release is made, Arch Linux will minimize the impact by performing all updates in its [testing] repository and only push updates on a yet to be decided day and time of the week. A news post will be made when that is decided.

Of course, all this makes systemd a hard dependency of pacman. We felt this was acceptable given Arch Linux has officially switched to using systemd. As this release is not tested (and unlikely to work) on systems without systemd, Arch users or other distributions using pacman will be required to make the switch to systemd if they want to continue using pacman as their package manager. The integration with system will become tighter in pacman-4.2 where we plan to use the upcoming kdbus message passing interface – through libsystemd-bus – to allow other programs to interact with pacman, making the development of alternative front-ends easier.

In terms of output, there has been improvements in a couple of areas. First colour support was added. This had been floating around for a long time, but no-one had ever spent the time to create a patchset and submit it. I think the colours for a simple update look good, although those when searching are a bit… rainbow. This can be only configured on or off at the moment. Extra informational output has been added for optdepends, providing details about whether an optdepend is installed or not and giving a warning when removing a package that is an optdepend for another. This also provides the groundwork for more complete optdepend handling in future releases.

When building packages using makepkg from this release, information about all the files in the package is stored, including permissions, modification times, sizes and checksums (md5 and sha256), etc. These can be checked using “pacman -Qkk“, excluding checksums (which requires additional support to be added to libarchive in order to read them in). Other useful features include never overwriting .pacsave files, but instead giving them a number suffix as needed. We have also polished the package signature checking, improving key importing and allowing configuration on how to validate packages installed with “pacman -U“, both using local files and from remote sources.

There are a few improvements to package building too. I have covered support for VCS packaging in makepkg previously, with bzr, git, hg and svn packages just requiring an appropriate line in the source array. Also a pkgver() function can be added to automatically update the pkgver variable in the PKGBUILD. With these VCS source lines, or any other source that is volatile, the value “SKIP” can be used in the checksum array.

An optional prepare() function can now be used in a PKGBUILD for preparation of the sources, such as patching and sed alterations. This function is run after the extraction of the sources and not run when --noextract is used, allowing operations that should only ever been run once on the sources to be skipped. Finally, a new debug option is available that will result all the debug symbols that are stripped from binary files to be stored in a separate package, which can be installed to allow easier debugging (another feature that has had patches floating around for a while).

Finally, two new helper scripts have been added to the contrib section: checkupdates and updpkgsums. The checkupdates script allows you to safely check for package updates without altering the system pacman remote databases. The updpkgsums script will perform an in place update of the checksums in a PKGBUILD, although more complex PKGBUILDs (such as those with different sources for each architecture) will not likely work…

So a long post, but this is a big release! There are enough of running the git version that it should be completely bug free, but just in case I am wrong report any issues to the bug tracker.

Edit: Yes – some of this was April Fools… (moderated comments are now restored too).