Changes To VCS Packaging Support In Makepkg

The current support from building packages from version control systems (VCS) in makepkg is not great for a number of reasons:

  • It relies on obscure (but documented…) variables being specified in the PKGBUILD, which actually achieve nothing in terms of downloading and updating the source as needed.
  • The whole VCS checkout/update mechanism needs repeated across every PKGBUILD that uses it so is a lot of unnecessary code duplication.
  • Building a package from a specific revision/branch/tag/… required using an altered version of this code, resulting in many non-standard work-arounds being made.
  • The automatic updating of the pkgver happens in what may not be an obvious way. For example, the pkgver for git PKGBUILDs is set to the build date, not the date of the last commit. Even if it was the date of the last commit, that can be far from unique. (Why not use git --describe? Because that relies on the tag being something suitable for an actual version number and many repos do not follow this.)
  • Even when a revision number is used for the updated pkgver, this results in different behaviour for different VCS. For example, with hg repos, you have to download/update the repo to determine the latest revision.
  • The updating of the pkgver is done before the makedepends are installed, so can fail if it relies on VCS tools.
  • The --holdver flag stopped the pkgver being updated, but the VCS repo was still updated to the latest version as usual.
  • VCS sources ignore $SRCDEST
  • Offline building (using pre-downloaded sources) required adjusting the PKGBUILD
  • You can not create a source package with the VCS sources included using --allsource

In fact, the issues with the current VCS implementation accounted for almost 10% of the bugs in the pacman bug tracker and there are a number more in the Arch bug tracker about how to improve the supplied prototypes for the VCS PKGBUILDs. It was clearly time for a rewrite.

An idea that had seen some discussion over the years, was to just put the VCS sources in the source array. Makes sense… right? The problem was choosing an appropriate syntax for the URLs that was consistent with what was already used and also flexible enough to handle the various possibilities of a VCS source. The format decided on is:

source=('[dir::][vcs+]url[#fragment]')

Simple! Well, it will be once I explain the parts… The url component should be obvious. The problem with it is that there is often no way to tell that is a VCS source. For example, for git repos without the git protocol enabled on the server, this will start with (e.g.) http://. To work around this, an optional vcs prefix can be added to the URL. So for git over http, you would used git+http://. This is based on the already used syntax when downloading subversion repo over ssh.

At the end of the URL is an optional #fragment. Providing information in a URL after a # character is some sort of standard that I am too lazy to provide a link for… Anyway, it allows us to specify information about what we want to check out when building. For example, I build my pacman-git package using the working branch of my git repo. To check that out, I use:

source=('git+file:///home/arch/code/pacman#branch=working')

Note the use of the git+ prefix there that allows me to check out from a local copy of my repo. The list of recognized fragments is built into makepkg and is documented in the PKGBUILD man page.

Finally, there is the optional dir:: prefix. This allows the specifying of a directory name for makepkg to download the source into. If not specified, makepkg trys to pick a good name from the URL, but there is such variation in VCS URLs that it will be often useful to change it. This is an old, but little known, syntax available in PKGBUILDs, which can be used to rename any source file once it is downloaded.

So now that VCS sources can be used, even multiple different repos to build the one package, how does makepkg chose how to update the pkgver variable? Sort answer is that it doesn’t. You can provide a pkgver() function that outputs a string to be used for the updated package version. This is run after all the sources are downloaded and (make-)dependencies are installed. For my pacman-git package, I use something like:

pkgver() {
  cd $srcdir/pacman
  echo $(git describe | sed 's#-#_#g;s#v##')
}

Currently supported protocols in the master git branch of pacman are git (branch, commit, tag), hg (branch, revision, tag), svn (revision). That covers ~92% of the VCS PKGBUILDs in the AUR. Adding support for the remaining VCS that are used (bzr, cvs, darcs) – or any other VCS – is quite simple but requires knowing how to efficiently use the VCS tools. I will create a patch to support any additional VCS if someone provides me:

  1. How to checkout a repo to a given folder.
  2. What url “fragments” need supported for that VCS.
  3. How to create a working copy of the checked out repo (i.e. “copy” the primary checkout folder) and how to get it to the specified branch/tag/commit/whatever. That can be in all one step.

Note that the old VCS PKGBUILDs will not stop working as such, although they are likely to be broken… At least the pkgver will no longer update. I’m sure there are other subtle incompatibilities too and you would still suffer from all the issues listed above, so it is definitely worth getting proper support for your needed VCS into makepkg.

If you want to take the new implementation for a spin, checkout a copy of the pacman git repo and build it. For those that are somewhat brave, you could even use the pacman-git package in the AUR, but make sure you know the risks involved in running a developmental version of a package manager entails…

One thought on “Changes To VCS Packaging Support In Makepkg

  1. Nice consistency improvement, will make VCS PGBUILDS more maintainable. Keep up the good work!