After a long time of planning and a massive rebuild, the “Great Python Rebuild of 2010″ has finally made its way into the [testing] repo for Arch Linux. And I do mean a massive rebuild… weighing in at 587 (and still growing) packages, that is a little more that 10% of our repos! That is bigger than the combined libjpeg/libpng rebuild earlier this year.

So what exactly has happened? Two things. Firstly, the python package now is from the python-3.x branch (currently 3.1.2). That means /usr/bin/python will point to /usr/bin/python3 being the default python-3.x interpreter. A new python2 package has been introduced containing the latest from the python-2.x branch (currently 2.7.0), with its primary binary being /usr/bin/python2. Note that also is an upgrade from python-2.6.5 which is currently in [extra]. While we were rebuilding everything anyway, I also enabled UCS4 (unicode) support in python2.

WHAT!!! Am I crazy? Python-3.x as the default /usr/bin/python? Well, my sanity may be questionable but the principles behind this transition are sound. Arch Linux is unashamedly bleeding edge so we do big transitions like this first. Other distros can follow in our wake. Also, python-2.7 will be the last in the python-2.x release series so we are following upstream intent and moving the focus to the python-3.x series.

Doing this transition now also allows us to gradually rename our packages. e.g. python-foo is still a package for python2 and will eventually need renamed to python2-foo. That would be much more difficult if we waited until we had a heap of (e.g.) python3-foo packages in the repos. Not being able to make a clean break in package naming like that is one of the disadvantages of a rolling release distro.

For all packages in the official repos, this transition should be very smooth. Packages from the AUR may need to be rebuilt to point at the /usr/bin/python2 binary rather than /usr/bin/python. A fair portion of packages will recognise the binary is named python2 and make all the required adjustments. Others will recognise the environmental variable PYTHON=python2 and use that. Finally, some packages will need a sed line to adjust scripts like:

sed -i -e "s|#![ ]*/usr/bin/python$|#!/usr/bin/python2|"
       -e "s|#![ ]*/usr/bin/env python$|#!/usr/bin/env python2|"
  $(find $pkgdir -name '*.py')


Of course, your own python scripts will need to point at /usr/bin/python2. However, by doing this you may run into portability issues across distros. There does not appear to be an easy solution for that at the moment. It seems that while most (all?) distributions include a /usr/bin/python3 link to their python3.x binary, none do the same thing for python2.x. Either create your own symlink in your path for those distros or even better file a bug with them asking for such a symlink. They are going to need one in the future…

If you are using the [testing] and [community-testing] repos go ahead and upgrade to the future of python. If see any of the following in one of our supported packages, report a bug:

• files linked to libpython2.6.so.1.0
• files with a “#!/usr/bin/python” or “#!/usr/bin/env python” shebang
• files in /usr/lib/python2.6 or /usr/include/python-2.6
• depends/makedepends/optdepends on “python” instead of “python2″
• packages that no longer work…

So far it looks like the rebuild has gone remarkably smoothly. A big thank you to all the developers and trusted users who helped out with the rebuilds. It also shows the power of our new repo setup on the mirrors. Now moving all those packages between repos just moves some symlinks so syncing to mirrors is very fast. So your favourite mirror should sync very quickly unlike with big rebuilds in the past.

But there is no rest for the wicked… We will be onto rebuilds for db and heimdal next!