Big Python Transition In Arch Linux

After a long time of planning and a massive rebuild, the “Great Python Rebuild of 2010″ has finally made its way into the [testing] repo for Arch Linux. And I do mean a massive rebuild… weighing in at 587 (and still growing) packages, that is a little more that 10% of our repos! That is bigger than the combined libjpeg/libpng rebuild earlier this year.

So what exactly has happened? Two things. Firstly, the python package now is from the python-3.x branch (currently 3.1.2). That means /usr/bin/python will point to /usr/bin/python3 being the default python-3.x interpreter. A new python2 package has been introduced containing the latest from the python-2.x branch (currently 2.7.0), with its primary binary being /usr/bin/python2. Note that also is an upgrade from python-2.6.5 which is currently in [extra]. While we were rebuilding everything anyway, I also enabled UCS4 (unicode) support in python2.

WHAT!!! Am I crazy? Python-3.x as the default /usr/bin/python? Well, my sanity may be questionable but the principles behind this transition are sound. Arch Linux is unashamedly bleeding edge so we do big transitions like this first. Other distros can follow in our wake. Also, python-2.7 will be the last in the python-2.x release series so we are following upstream intent and moving the focus to the python-3.x series.

Doing this transition now also allows us to gradually rename our packages. e.g. python-foo is still a package for python2 and will eventually need renamed to python2-foo. That would be much more difficult if we waited until we had a heap of (e.g.) python3-foo packages in the repos. Not being able to make a clean break in package naming like that is one of the disadvantages of a rolling release distro.

For all packages in the official repos, this transition should be very smooth. Packages from the AUR may need to be rebuilt to point at the /usr/bin/python2 binary rather than /usr/bin/python. A fair portion of packages will recognise the binary is named python2 and make all the required adjustments. Others will recognise the environmental variable PYTHON=python2 and use that. Finally, some packages will need a sed line to adjust scripts like:

sed -i -e "s|#![ ]*/usr/bin/python$|#!/usr/bin/python2|"
       -e "s|#![ ]*/usr/bin/env python$|#!/usr/bin/env python2|"
  $(find $pkgdir -name '*.py')

Of course, your own python scripts will need to point at /usr/bin/python2. However, by doing this you may run into portability issues across distros. There does not appear to be an easy solution for that at the moment. It seems that while most (all?) distributions include a /usr/bin/python3 link to their python3.x binary, none do the same thing for python2.x. Either create your own symlink in your path for those distros or even better file a bug with them asking for such a symlink. They are going to need one in the future…

If you are using the [testing] and [community-testing] repos go ahead and upgrade to the future of python. If see any of the following in one of our supported packages, report a bug:

  • files linked to libpython2.6.so.1.0
  • files with a “#!/usr/bin/python” or “#!/usr/bin/env python” shebang
  • files in /usr/lib/python2.6 or /usr/include/python-2.6
  • depends/makedepends/optdepends on “python” instead of “python2″
  • packages that no longer work…

So far it looks like the rebuild has gone remarkably smoothly. A big thank you to all the developers and trusted users who helped out with the rebuilds. It also shows the power of our new repo setup on the mirrors. Now moving all those packages between repos just moves some symlinks so syncing to mirrors is very fast. So your favourite mirror should sync very quickly unlike with big rebuilds in the past.

But there is no rest for the wicked… We will be onto rebuilds for db and heimdal next!

13 thoughts on “Big Python Transition In Arch Linux

  1. Thanks, but what is about PyQt. Will PyQt support Python 3 by now, or do i still have to install it from AUR?

    • That depends on the maintainer of pyqt and what they decide to do with the package.

  2. Pingback: Python 3 et Gwibber ? Une vraie galère. | Le Weblog de Frederic Bezies

  3. Hello Allan,

    Thanks for the hard work! In my case, as a were-user of [testing] and a Python developer, I upgraded my box with new py packages. The transition was quite well as you said but most of the installed python applications under /usr/lib/python2.6/site-packages/ have stayed there and therefore weren’t runnable. For instance, when I reinstalled ipython, it was installed under that directory but not py3k’s site-packages folder. Is this because it wasn’t rebuilt? I thought maybe it could be related with setuptools or something?

    • You need to enable the [community-testing] repository to get rebuilds for [community] packages.

  4. > It seems that while most (all?) distributions include a /usr/bin/python3 link to their python3.x binary, none do the same thing for python2.x.

    Not sure what distributions you have checked on, but on a few boxes I have access to ($ cat /etc/issue | head -n1; whereis python2 | trim useless shit):

    Red Hat Enterprise Linux Server release 5.5 (Tikanga)
    python2: /usr/bin/python2.4 /usr/bin/python2

    CentOS release 5.5 (Final)
    python2: /usr/bin/python2.4 /usr/bin/python2 /usr/bin/python2.6

    Debian GNU/Linux 4.0 n l
    python2: /usr/bin/python2.4 /usr/bin/python2.5

    So it looks like just Debian sucks. :)

    • You are right. It appears it is just Debian. Ubuntu even has it.

      I was sure I tested a Fedora box I had… turns out that did not have python installed on it at all! So not the best of testing…

  5. > files with a “#!/usr/bin/python” or “#!/usr/bin/env python” shebang

    can you expand? What’s wrong with that (if the package if supposed to work with python 3) ?

    • If it is python3 compatible code, and it only depends on python3 compatible libs, then nothing.
      But most of the time #!/usr/bin/env python3 is used if it is…

  6. I appreciate the courage in doing such a transition, now I have an excuse to produce python3-compatible code :)

  7. The move to python3 was 1+ year too early!

    There’s no Python Web Framework which works with Python3, because there’s no WSGI spec. for Python3. No Django, Pylons, Zope or whatever! :(

    Sure I can use the python2 package and the switch is flawless, anyway move so early just cause more trouble then move in 1 year.