I have deleted the [allanbroke] repository. It was started mainly to test the PGP signing implementation in pacman, which is now well established. Also, I would delay any packaging of release candidates or beta releases for this repository until I had enough free time and often official releases were made before that happened.

The repository may return someday, perhaps with VCS builds of packages I use locally once I get around to automating their creation as that would require no extra work…

I was recently invited to give a talk about Arch Linux at SINFO XX at IST in Lisbon, Portugal. It was a whirlwind tour of Europe, with the time I spent in transit almost exactly equal to the time I spent there.

Check out the video of my talk on their YouTube channel. I discuss what makes Arch different from other Linux distributions, what our strengths are as a distribution and briefly cover what future plans people have. I’m not going to watch it as it is never a good idea to dissect talks too much, so I’ll just assume I was awesome… It was also after midnight in my time zone, so I blame any mistakes on that.

Thanks to the organizers for inviting me over!

Edit: Quite a few people have asked for copies of me slides. Here they are (CC BY-SA): ODP PDF

According to Distrowatch, there are eleven “distributions” “based” on Arch Linux. I use “distributions” in quotes, because some are far less of a distribution than others and “based” gets quotes because some are so based that they are really Arch Linux in a poor disguise…

I have seen a bunch of new release announcements for several of these distributions in the last few days, so I thought I would take some for a spin and see what I am missing out on.

All distributions were tested in VirtualBox with a 128GB disk and 2GB RAM.

Contender #1: Chakra – 2012.12 Claire

The Chakra Project is my #1 contender as it is a real Linux distribution. They do use our PKGBUILDs (sometimes too directly…) but they rebuild everything for themselves. Chakra also has quite an interesting take on the rolling release model, opting for a “stable” core of the distribution and rolling release userland. So toolchain updates are rare, but the newest KDE will be packaged within days. I have issues with their “stable” core, because stable means no updates at all. Even minor bug fix updates are not considered and I am sure there is no backporting of any security fixes either as their team is too small to handle that. For example, I would not be running pacman-4.0.2 or even an unpatched pacman-4.0.3 as my distributions package manager. Another thing unique to Chakra is the pure KDE set-up. Any popular software that is not Qt based (e.g. firefox, GIMP) is supplied in a bundle. I find these a very bad idea, as many of the same libraries are on your system multiple times but in separate bundles, and it also appears that these are easy to break and under-maintained.

Onto the installer… Chakra wins points for being the only pure graphical installer out of the distributions I consider here. I did a review of the installer early in 2010 and there seemed a complete lack of improvement. The Live DVD booted fine and welcomed me with a window that pointed out the difference between Chakra and Arch, and noted that pacman is not designed for a GUI frontend (debatable…) so it will be replaced one day. Starting the install gives three screens worth of install notes which no-one has ever read and if anyone did read them, I am confident they did not care that their website is being redesigned. The partitioning took me to an external tool (KDE Partition Manager), which I actually found rather difficult to use. It took me several attempts to get everything set so that the installer would let me continue.

Once I finished the install, I rebooted… to a prompt. Whatever autodetection is done to get the KDE session up a running post-install had failed in VirtualBox. So no further review was done.

Contender #2: CinnArch – 2012.12.21

I had heard good things about the Cinnamon desktop, so CinnArch was my next choice. This distribution is Arch Linux – when I look at the package information for glibc, I see that I packaged it. The only difference is that an additional repo is provided with the packages needed for Cinnamon desktop to run.

The installer boots to a nice live system. There is two installation options – “CLI installer” and “Graphical Installer” but the latter is not selectable so “CLI” it is… In this case, CLI means an ascii-graphical menu driven installation environment that was easy to follow. The only steps that I found amusing was finding the fastest mirror to install from, which (with only slight exaggeration) took longer to complete than downloading the packages from the slowest mirror would have. And then on with installing the packages. Or not… It turns out that the installer detects I am using VirtualBox and tries to install the virtualbox-guest-modules package, which currently has a broken dependency and so can not be installed. There is no way to skip this that I could find.

So lets play around in the live media to see what might have been. It turns out that CinnArch provides two repos: cinnarch-repo and cinnarch-core. I could not find the justification for requiring a second repo. The repo looks fairly good on a quick glance, with all packages signed and a keyring provided. Theoretically, you can just enable that repo and install cinnamon from there, but I could not find where to see a list of keys that the developers use that would enable me to install the keyring package and then verify the rest of the packages. One thing I was concerned about was the filesystem package that was being provided for reasons I could not ascertain. The biggest issue with providing a repo for Arch Linux is that you need to keep up with our rolling base. That means your repo needs to have rebuilds available when we push (e.g.) a library update with an soname change. Providing an unnecessary filesystem package seems to only make things difficult for them. This was also my first time seeing the Pacman-XG GUI for the pacman package manager. That is enough said about that…

Contender #3: Manjaro – 0.8.3 XFCE edition

I was not expecting much from Manjaro given I had read this review of a previous installer. But given the “success” of the last two distros, it has a chance… Some things from that review had not changed – in the live environment there is still the unnecessary password of “manjaro” for both the user and root. And I was not endured by the green colour scheme or the logo which is used for the “start” menu. But to give Manjaro credit, I successfully installed using the old style Arch ascii-graphical install, which was easy for those that are use to that. Reiser3 as the default filesystem still confuses me.

So what do you get from Manjaro? It seems to be Arch delayed plus extras. The idea is that the Arch repos are delayed reaching you for a few days while everything is sorted out to give you an easy upgrade. This happens by adding a package manjaro-system which is always upgraded first and runs a script that automatically handles any manual intervention that would be required on a pure Arch Linux system. As a heads up, this uses a feature of pacman called SyncFirst that is removed for the upcoming pacman-4.1, so they may need to rethink their entire system soon.

The first difference I noted was it ran a 3.4.24 kernel, so that have held that back from being updated to the newest release, but at least it is the newest 3.4 series release. Looking at the glibc package once again, I am the packager, but it is a repo called [platform]. The Arch [extra] and [community] repos appear to be used as-is, so the need for [basis], [platform] and [addon] repos to replace our small [core] repo is strange. Also, this was the first time I had ever heard of a GUI for pacman with the imaginative name pacman-gui. It allows you to run pacman -Sy, which just refreshes the databases – a crazy thing to do without immediately updating in a rolling release distribution that does not provide multiple versions of a library. And it is not much of a GUI either, as it just launches a terminal that runs the pacman command you just clicked. There was also a LibreOffice installer that called pacman in the background.

The Winner?

Sadly, I do not think there was any winners today. Chakra had the most polished live environment, but failed to boot to the desktop. CinnArch failed through no fault of its own, which is sort of a fault of its own for not being a real distribution. Manjaro installed, although I saw nothing to make me want to recommend it.

The may be the most painful statement I have ever made… but ArchBang might be the winner!

Edit: Twelve hours later and the bug on the Arch end preventing the CinnArch install has been fixed. CinnArch installed and booted to the desktop without an issue.

It seems I have gained a “fan” or two over the last week. But this guy is my current favourite. Shutting down the arch-general mailing list for a day resulted in me getting the following email:

Subject: wrong move
Date: Wed, 26 Sep 2012 15:16:03 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Very bad move .

You will just inflame it for later if you have not learnt that yet what
are you doing running a mail list .. ?..

Pete .

Thanks Pete. Lets see who gets angry at me the most…

Subject: List admin
Date: Thu, 27 Sep 2012 10:42:44 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: arch-general-owner@archlinux.org

I wish to know who is the main arch-general@archlinux.org admin

i wish to start a vote to remove Allan McRae
or at least demote him . He appears to be prone to panic attacks and
ill judged moves ie shutting the list down ..

Before anyone says anything else i run 4 lists myself , all motor sport
related .

The other point is if certain people would just answer the question
instead of going off on some tangent it would help lessen the wars

You may not agree that is not my problem i see what i see this list
needs someone that does NOT panic

Pete

It would be a fair assumption that if I have the ability to shut down an email list, then I might have administrator access… A lack of reply and this email arrives:

Subject: Mail owner
Date: Mon, 1 Oct 2012 08:26:01 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: arch-general-owner@archlinux.org

I am once again calling for the removal of Allan McRae as an
unsuitable person to be in control of a mailing list ,

He is Childish despotic offensive officious obnoxious and is
stomping around like some little Hitler .

He has go to GO

Pete

The use of “despotic” was particular non-inventive given that was already used on the arch-general mailing list earlier that day. So the Godwinism was the best that could be done.

I subtly mentioned these emails in a reply to arch-general in order to let him know that I have seen them. That resulted in:

Subject: Un warrented transference of mails to list
Date: Mon, 1 Oct 2012 19:07:26 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Excuse me .

I kept OFF the list for a reason , you are in a flat out panic spin
this shows a mile off by your actions .

If you are unwilling to run the list and let it run then i suggest
you retire and let someone else do the job .

You are getting as bad if not worse than some of the bird brains on
the suse lists that is the reason a good number of new users have
appeared here Like me they were getting sick of the blatant junk from
the list owners .

This mail was sent OFF list for a reason get with it .

Oh and i still say RETIRE if you cant handle it ..

Pete

If you did not want emails to be public, perhaps you should not have sent them to an email address that you had no idea who the recipient was. Or, even better, just do not send them at all. I actually made an exception to not answering any of these emails and replied with that friendly piece of advise.

But that was not the end of it…

Subject: tosser
Date: Wed, 3 Oct 2012 08:53:15 +0100
From: P .NIKOLIC <p.nikolic1@btinternet.com>
To: Allan McRae <allan@archlinux.org>

Allan

you are by far the biggest silly little boy i have ever had the
misfortune to stumble on.

How the Arch Linux community ever managed to get a twat like you as
the mailing list owner i will never know but hey there will soon be a
new mailing list .

Ban my aunt fanny word of warning WATCH your list you
think you have stopped me THINK AGAIN 🙂 … Oh and IP filtering
will get you nowhere at all

have fun

Pete .

I feel sad for his inability to form good insults. Using “biggest” and “little” together shows a poor grasp of the English language. This would be fine for non-native speakers, but from his email address I conclude that he is in the UK and given there is a person who dislikes immigrants with the same name on Google+ (who has me in a circle…), I will assume he is a native English speaker too.

Which finally gets me to the point of this post. It appears Pete will soon be creating a mailing list where people can rant about Arch Linux and the direction it is moving all they want with no fear of being censored. Thanks Pete!

For those who came here looking for the solution to the systemd “problem” in Arch Linux, this is not the article you are looking for. I care very little about my init system beyond that it should successfully boot my computer and start the software I need it to start. In fact, my entire understanding of the boot process goes “Push button… *MAGIC* …prompt”, so taking advice from me on the boot-up process may not be the best idea…

However, what I do know about is the Arch Linux packaging system and how to put together a Linux distribution. So I am going to discuss how people would go about providing all the tools to run an init system that is not systemd in Arch Linux. Many lessons will be taken from how systemd was provided in Arch Linux; firstly as a community based projected and then as official packages providing a secondary init choice in the repos. From what I see from current efforts, people seem slightly naive about what is required and are completely ignoring what has gone previous.

Lets start with choice number one. Where are you going to get udev from? I see two choices here. Firstly, you could just create a package containing on the udev files from the systemd tarball. This is the approach used by Linux From Scratch and you could even use their Makefile. The original email announcing the merge of udev into systemd states that udev can still be built for usage outside of systemd systems and that will be supported officially. So I personally would choose that option. However, I know people are concerned that udev will become more fully dependent on systemd. Here is the email that people cite as the end on non-systemd udev. I read that as saying that genuine issues when using udev outside of systemd will be fixed. There is also nothing saying that patches for udev that do not impact its usage in systemd will not be applied. Anyway, for those people there is a fork of udev being developed. If you select this option, there are a couple of things to be aware of. Firstly, there is is no guarantee of compatibility with the udev from systemd so the libudev. In fact, the fork has a different soname and that means you will need to recompile all software that links to it (~30 packages in the Arch repos). Secondly, the development of this fork currently appears to be porting relevant commits from the systemd tree to a snapshot of the udev codebase before the merge happened. It will be of interest to see what happens as these code-bases diverge and whether any independent development (excluding the build system…) occurs in the fork. (See Edit #2 below)

Choice number two is what you are going to do with software that links to the various libsystemd-* libraries. The anti-systemd way is to rebuild all this software to not have this dependency. That is only 12 packages at the moment, although the number is growing… An approach requiring no rebuilding would to just provide a libsystemd package. These are just libraries on your system and most software that uses them has something like if(sd_booted()) in it. That results in these doing nothing when your system was not booted with systemd.

If providing separate packages for udev and libsystemd, you will need to be careful with the provides array in their PKGBUILDs so that you avoid unnecessary rebuilds. But speaking of unnecessary… Why provide separate packages for udev and libsystemd at all? Just have the systemd package installed. If you do not boot with systemd, then its binaries just sit around on your system doing nothing but take a whopping 10MB of space (that number is pure guesswork…). Call me lazy, but all that packaging seems a lot of effort. Do you repackage libjpeg-turbo to get rid of its binaries that you do not use?

Once you have managed to get that far, you will need to figure out which init system you will use and how you will manage services. I am going to state that the current Arch Linux initscripts are a dead end. The current version requires systemd binaries, so to avoid that you would need to grab an old release. Then you would have to fix the bugs that were fixed by moving to the use of systemd binaries. Finally, when Arch Linux moves to systemd by default, the service management scripts in /etc/rc.d will gradually be removed, which is no great loss as they are horrid anyway. I will also assume that some monitoring of services and restarting them when they fail is a worthwhile goal, as no-one really has spoken out against that. Completely ignoring upstart (because yuck…), you have two real options for this.

Option #1 is to use runit. This is a complete init system replacement with service management and seemed to be a popular choice among people with severe systemd allergies on the arch-general mailing list. As a bonus, the website has a whole heap of run scripts that you can use to provide a runit-arch-services package (analogous to the systemd-arch-units package that was provided in the early systemd days and is currently being phased out). Note that means you are going to have to provide an runit run file for every piece of software in the Arch repos if you are doing a proper job. If you are going to use runit, you may want to also consider the ignite project which provides additions to runit that allows it to use some old style /etc/rc.conf style configuration. I am going to pretend that ignite‘s definition of “adapted from Arch initscripts” does not mean GPL copyrighted code was copy-pasted and released as public domain!
(See Edit #1 below)

Option #2 is to use OpenRC, which is developed by people over at Gentoo. This works with your systems init, so you are going to need to keep sysvinit as well. Because it is compatible with the Gentoo init scripts, you will probably be able to find the daemon scripts you need in the Gentoo portage tree. I assume for a more “old school” Arch Linux experience, you would want to do something similar to ignite and pilfer relevant bits from the Arch Linux initscripts package. I believe on of the openrc developers posted to the arch-general mailing list suggest it was used, so there is probably help available getting this running for those that are interested.

That is a brief summary of what is required to purge your system of the “systemd virus” – for now. But as a virus, it will spread… Soon you will need to deal with a lack of ConsoleKit and if all the hysteria is to be believed, every single other piece of software on your system will soon be absorbed into systemd. So if you are going to do a lot of work to avoid systemd, be prepared for the amount of work to increase in the future. But do not concern yourself too much… Arch Bang has plans to save all the systemd haters! And they have made an installer so their ability to handle actual packaging of something this complex is not to be questioned!

Post publication edits:
Edit #1: I am removing my comment about the licensing of ignite for two reasons. 1) The author contacted me and pointed out the only directly copied bits was the mountpoint code (~4 lines). I see other similarities, but admittedly there is not many other ways to do [[ $foo ]] && ... style tests. 2) There is no license in the Arch Linux initscripts code base. It used to print “GPL2” when booting, but that has even been removed.

Edit #2: Do you know how to fix the udev fork having a different soname for its library? Use this patch! Now I am convinced the author of the fork knows too little for it to be useful.

I hear rumours that Arch Linux is probably switching to systemd as its default init system at some stage. Or at least that is what has been reported on by some Linux “news” sites based on a post to the developmental mailing list proposing we start the migration. I personally would have waited for something more official before reporting, especially because there is a lot that needs done before that can happen.

Anyway… with that upcoming change, I decided it was time for me to finally install systemd and see how hard the transition would be. The answer? Not that hard at all, although I made it rather difficult for myself as you will read.

I figured the first step in moving towards using systemd was to deal all the configuration files. I tend to ignore updating configuration files unless something stops working, so my /etc/rc.conf was full of LOTS of old settings. Following the archlinux(7) man page, I altered /etc/hostname, /etc/locale.conf, /etc/vconsole.conf and /etc/timezone and removed all their parts from rc.conf. I also deleted the network section (which has been useless forever given I use NetworkManager) and the modules section (as the modules are all autoloaded these days). That left the DAEMONS line. I rebooted to see whether everything was still OK at this stage and I struck failure #1. The console went all funny during boot-up, changing characters into weird symbols and losing colour. So my vconsole.conf was obviously bad. From its man page: “FONT= defaults to latarcyrheb-sun16“. What I did not realize was that means you still have to add a FONT= line.

Now to actually install systemd. A simple pacman -S systemd, adding init=/bin/systemd to my grub kernel line and I was ready to go. I want my nice looking LXDM at boot-up, so I enabled that service with systemctl enable lxdm.service. And reboot!

Booting when fine, I logged in and everything was happy… which confused me. I had not enabled all the other services to replace the daemons I start in rc.conf. It turns out that some work has been done by the relevant Arch Linux developers to help people transition to systemd and it will still load modules and daemons specified in rc.conf. But I want a pure systemd boot-up so I need to disable that. Unfortunately, I could find no sensible way to achieve that. I could either comment out DAEMONS from rc.conf – but that makes reverting to using the old initscripts require more than altering the kernel line – or I could be heavy handed and delete the service files from the initscripts package. I did the latter.

So systemd booting attempt #2. Prior to rebooting, I enabled the NetworkManager service as I like having access to the internet while playing with my system. Reboot and all is fine. Sort of… I have internet access, so Network Manager is running, but where is my applet? I need that applet to easily access the range of networks my laptop goes on and VPNs etc. My strategy for solving things like this goes: #1 – run from a terminal and see if there is an error message; #2 – run as root and see if that works. Running nm-applet as root worked. I figured this was maybe LXDM not being completely systemd friendly yet and something not being launched properly, so I edited /etc/dbus-1/system.d/org.freedesktop.NetworkManager.conf and gave my user more permissions. And all is fine again…

Until I try getting sound. I was in the middle of watching Fight Club when I embarked on this adventure and despite having seen it before, sound is still important. Looking in the alsa package file list I see a couple of service files. I try starting them and I get a message: They are not meant to be enabled using systemctl. Lets try running alsamixer – nothing – and as root? It works… Poop.

Now remember how I said that I do not deal with configuration file updates. My system is littered with .pacnew files. Obviously I am missing permissions due to logging in with LXDM. And sure enough, /etc/pam.d/lxdm had an update and that involved adding pam_systemd.so. Sounds important. Fix that, revert my changes for NetworkManager and reboot. Internet and sound. Great success!

On to enabling all my other daemons. No need to deal with dbus or netfs as they are handled automatically and the new logging is fine by me so syslog-ng can go. I enable ntpd.service and then attempt to start crond.service… Fail. Looking in the cronie package, I see a cronie.service file that works. Strange thing is, there is a crond.service file and it is a symlink to the cronie.service. I have not idea why the symlinked version can not be enabled, but I strike the same thing again for cupds.

The only other daemon left is my custom fan control. For that, I need to write my own service file. It is a very simple daemon, so has a very simple service file:

[Unit]
Description=Allan's MacBook Pro Fan Control Daemon
 
[Service]
ExecStart=/usr/bin/mbpfan
 
[Install]
WantedBy=multi-user.target


Much better than the 70 line monstrosity for the the old rc.d script.

The final daemons I used to launch manually were php-fpm, mysqld and nginx to run WordPress locally when I want to draft a blog post. I heard that you could easily customize service files so I wanted to add a dependency on php-fpm and mysqld to nginx so I only need to launch one thing. That was as simple as creating /etc/systemd/system/nginx.service with:

.include /usr/lib/systemd/system/nginx.service
 
[Unit]
Requires=mysqld.service php-fpm.service


Everything was running, so now it is time to remove initscipts and sysvinit. And a final reboot to ensure everything is still fine… but I can not longer shut down from within XFCE. Of course, removing sysvinit gets rid of the shutdown binary, but that is replaced by installing systemd-sysvcompat. That also provides a /sbin/init symlink so I can remove the addition to my GRUB kernel line. Finally reboot and almost all is fine. I am a picky bastard and so could not accept there being some bootup output not cleared from my tty1. Adding “quiet” to the kernel line removes all that output and I am happy.

I have not been using systemd long enough to give it a proper review. But I can say it works, at least for me. Also, shut down is now amazingly fast. I know that may not sound that important, but I do shut down my laptop in a rush at the end of the work day to run for the train. So saving time there is good for me. There was essentially no difference in boot-up speed for me. But NetworkManager is now fully up and running and connected to the network by the time I get logged into XFCE now. It is also what takes the most time to launch, so I wonder if that is holding up the process somehow. The only thing that annoyed me was not being able to disable the rc.conf parsing services, but I am informed there is a patch for that on the arch-projects mailing list. That should make it a lot easier for people switch back and forward between using SysVinit and systemd when doing this transition.

PS. I know I capitalized systemd in the title when it should not be… but I like my post titles with capitals.

With the recent “controversies” in Arch Linux – such as the move to /lib being a symlink, parts of the rc.conf file moving into separate files and the installation becoming command line based – I started thinking about what really defined Arch Linux in the early days and what make it what it is currently. To assess this, I read through all the news and interviews about Arch up until the 0.8 release. That covers about the first five years of the Arch Linux existence. Here is what I took from those articles, in no particular order:

Pacman
This is probably the most unique thing about Arch Linux and it is the reason I started using Arch in the first place. I may be ever so slightly biased here… but I still find this one of the best package managers around. This is not necessarily for the pacman manager itself, but the package building system is so incredibly simple. Let me be clear in pointing out that there are a lot of areas that I think could be substantially improved, but the major things I want implemented are gradually being crossed off my TODO list.

The Arch Build System (ABS)
Being able to readily customize packages is always touted as on of Arch’s best features in any review I see. I never use it… Well, not entirely true – I have 1 customized package out of the 626 currently on my system. I wonder if it is the packaging system rather than ABS that is being highlighted by not well informed reviewers there, because maintaining a custom package is really not a pleasant task in Arch.

Rolling release
Having packages available in our repos very quickly after the upstream release is a major attraction to Arch Linux. While Arch is far from the only rolling release distribution, the updates tend to be damn fast!

Optimized for i686
Ten years ago, most distributions were compiled for i486. So back then, compiling for i686 was a big feature. Now 64-bit has overtaken the world and and most 32-bit distribution offerings are built for i686, so this is not an advantage anymore. We should probably drop i686 and crank up the optimization of the x86_64 port!

Simple filesystem layout
I found this an interesting feature to be mentioned by Judd… Especially given all sorts of packages in Arch used to be shoved in /opt. But he was referring to the lack of things like /usr/doc. Years ago, Arch used to ship packages without documentation – including info pages. Now we do ship info pages (they are treated identically to man pages by the packaging system) and documentation is also shipped (in /usr/share/doc, not /usr/doc).

Simple configuration
The rc.conf file was highlighted by many reviews as being the one stop for all your configuration. In fact, one review suggested that it should also include “wi-fi profiles, printer, scanners, bluetooth etc” given there were not GUI tools to do this. The lack of distro specific tools for configuration was also highlighted, although that is a bit weird given our rc.conf is ditro specific.

Support
The help on the IRC channel, and to a lesser extent the forums, was often mentioned. Also the excellent documentation in the wiki was beginning to make its appearance.

Rapid adoption of new features
Having the latest an greatest software was often considered a highlight. For the initial five years of Arch Linux, that included things like the Ext3 and Reiser filesystems, udev support and Linux 2.6.

One thing I did not see mentioned anywhere was Arch’s use of vanilla packages. The minimal patching is one of my favorite features of Arch Linux, but I guess that is not something that would be readily seen by a reviewer.

So… lets take a look at the current controversies in Arch Linux land with respect to the foundations of Arch Linux.

Controversy #1 – Symlinking /lib to /usr/lib
People were not particularly happy about this. And they are really going to hate when /bin, /sbin and /usr/sbin all point to /usr/bin… Note there were quite a few other things that have become symlinks recently too (think about what points at the /run directory). The difficulties of the /lib change were exasperated by the rolling release model and limitations in pacman in performing the upgrade. But I think this fits in directly with the “simple filesystem layout” above. As discussed above, we have slowly moved away from this principle to provide packages closer to what their developers intended (by not removing documentation), so this is really a return to original Arch principles!

Controversy #2 – The demise of /etc/rc.conf
While the single rc.conf is highlighted as major feature of Arch Linux, reading the reviews makes you notice that configuration of an Arch install was never down to a single file. Other files mentioned included…

/etc/conf.d/wireless
/etc/fstab
/etc/group
/etc/grub.conf
/etc/hosts
/etc/inittab
/etc/lilo.conf
/etc/locale.gen
/etc/mkinitrd.conf
/etc/modules.conf
/etc/modprobe.conf
/etc/rc.local
/etc/resolv.conf
/etc/pacman.conf
/etc/X11/XF86config
~/.xinitrc

And that is just the configuration files mentioned in what were quite superficial reviews of Arch Linux. So a single configuration file was really just a myth.

But lets take a step back here… How about some quotes from Judd, the founder of Arch Linux:

“In Arch “simple” is different what other distros are considering. The learning is more important than getting something easily done.”

“Relying on GUIs to build/use your system is just going to hurt a user in the end. At some point in time a user will need to know all that some GUIs hide.”

What does a single configuration file get us? Is it “simple” in terms of the definition above? Not really… It requires that we write our own tools to deal with it and convert these into the appropriate commands during boot-up. For example, instead of configuring modules in /etc/rc.conf, we should directly use the configuration files that the module loading software reads. How is putting the settings in rc.conf and having software interpret them any different than using a GUI to do the same thing? It does not give us any actual understanding of the system. Also using the same configuration files as other distributions do (and this is becoming more standardized) means the learning done in Arch Linux will apply to many other distributions.

Controversy #3 – The move towards systemd
I think it is plainly obvious that systemd will become the primary system initialization method for Arch Linux in the not to distant future (not that this has been formally decided, so take that statement as unofficial…). Having not used systemd at all, I really can not comment directly on if it is any good or not. But what do we lose by moving to systemd in Arch? I have seen the following criticisms:

Users are being forced to use systemd. This argument really does not hold water. How many initialization systems are currently in use on Arch systems? Just the one currently “forced” on you…

The module control is more complex. Right… because “rc.d start foo” is much simplier than “systemctl start foo“. I think the real criticism people are trying to express is that the rc.d way of doing things is more traditional Arch. But have you looked at the quality of scripts in /etc/rc.d? It is generally poor. Moving to systemd unit files instead will be an advantage as they are much more simple to write and can be pushed to the upstream packages.

The initialization process will no longer be controlled by a bash script so it will be harder to skip a broken section. This is probably somewhat valid. I do not know enough about systemd to really comment beyond a couple of points. Firstly, systemd is (or will be) used by many distributions so will be less likely to break than our current initscripts. Also, systemd is quite modular, so it appears that stopping particular modules should be achievable.

It is written by Lennart Poettering. I have no idea how this ever became a “genuine” argument against systemd, but almost any discussion involving systemd ends up being about Lennart also writing pulseaudio. Maybe we need an updated version of Godwin’s Law for this situation… Any time an argument reaches this point, I know all things technical are out the door and I stop following.

When I was reading the old reviews, every time I saw mention of Arch Linux using the latest innovations in the Linux world, I started wondering why we are not using systemd already? Here I have only (partially) refuted the downsides of systemd, not pointed out what I can see are some quite substantial benefits. I personally think the historic rapid adoption of new features should continue in Arch Linux.

This has been a much longer post than I initially intended, but I hope this makes people think about what makes Arch Linux what it is. Or at least reconsider whether changes being made are really going against historical Arch principles. It has made me consider why I have not tried systemd yet, so I will do so in the next couple of weeks. Being on the bleeding edge is not much fun without the occasional cut…

NOTE: This post applied to the 2011.08 Arch Linux release. The current install ISO does not need the following steps, although they may still be useful for people attempting to upgrade an old install.

The move to /lib being a symlink to usr/lib in Arch Linux has been completely smooth… for those that followed the detailed instructions on the wiki. Not so smooth for those who had brilliant plans like deleting /lib and updating to the new glibc really quickly. What linker were you going to use to do that?

Anyway… it is now appears quite difficult to get a fully updated system from a core install. I say appears, because I did it in a couple of minutes. But I admit to actually knowing what I am doing! The simple solution is not to do a core install, which uses the packages that were available at the time of the installer creation. Instead do a net install and download the latest version of all the packages and be fully up to date straight away. Or, if you must do a core install, grab a newer testing install image.

If neither of these are an option, here are the steps required to update:

pacman -Sy
rm -rf /var/run /var/lock && pacman -Sf filesystem
pacman -S tzdata
pacman -U http://pkgbuild.com/~allan/glibc-2.16.0-1-i686.pkg.tar.xz
rm /etc/profile.d/locale.sh
pacman -Su --ignore glibc
pacman -Su


You might need to reboot after filesystem install. Also, change i686 to x86_64 in the pacman -U line if needed… Always say “N” to updating pacman first, and say “Y” to all the replacements.

All of that is given in the various news items about interventions required during updates since the last installer release. The only trick is getting the packages updated in the right order.

This is the price you pay for a rolling release distribution. Arch Linux does not get the benefit of being able to make changes like this at the time of a major release because we have none. We could probably automate some of these things with package update trickery… but it is our policy to not hide these things from the user. Finally, there is work being done on getting a new install image released more frequently so that upgrade paths in the future will always be simple.

Given the number of unpleasant emails I have received over the last few days, it appears that people have forgotten what the purpose of the [testing] repo in Arch Linux is for. Wait for it…. testing packages! Who would have thought!

Now to clarify some things. While the [testing] repo is for testing, we do our best not to break things in it. In fact, the [staging] repo was set up to allow rebuilds to occur away from the main repos and allow [testing] to always be in a usable state. Also, many (maybe even most) of the Arch developers use the [testing] repo on their main computer. I know I use it on all of mine, including on my work laptop. So I really do not want things to break either. And of course, if I cause breakage, that limits my ability to yell at other people who break packages.

One of the common criticisms I got in my “fan mail” was that I did not do enough testing of the change that saw the /lib directory change into a symlink to /usr/lib. Perhaps the need for testing was why I put the package in the [testing] repo… But, also I did do a fair amount of testing of this update. I tested that pacman stopped the update when there was a file in /lib (or in a subdirectory of /lib), both when the file was owned or unowned by a package. I checked that pacman -Syu --ignore glibc && pacman -Su worked from and old base install.

What I did not test was using pacman -Sf glibc to resolve a conflict, mainly because even with my low expectations on the general population’s intelligence, I did not expect Arch Linux [testing] users to be quite that incompetent. I also did not test having another package owning the /lib directory or a subdirectory within that folder but with no actual files in it (or having its files manually moved from in it). And I did not test upgrading from a more complete install that could have packages with versioned dependencies on glibc (which does not actually break the update but makes it a bit more difficult).

Now ignoring the usage of -Sf… the only case I did not test that actually causes breakage was an empty subdirectory in /lib or another package owning /lib with no actual files in there. That should never actually happen, but it appears people manually moved module or firmware files from /lib to /usr/lib without fixing thier packages, creating this situation. In both these cases, pacman would get to extracting the /lib symlink, see there was still a directory there and refuse to extract. That left people with a system looking for the linker in /lib, but it not being there. Annoying, but this situation is easily recoverable, even without a rescue CD.

You may ask why pacman did not detect those conflicts before attempting the extraction of the package. That would be a good question… It seems the situation of changing a directory to a file or symlink is not a very common operation and so was not very well tested in the (quite extensive) pacman test-suite. Also, a bug fix in pacman 4.0 (prevent removal of empty directories if they are owned by another package), will have caused this bug to be exposed much more frequently. A couple of patches to pacman and these conflicts are now caught prior to the upgrade taking place. These patches are now applied to our pacman package, so non-testing users will not be exposed to this issue.

So a couple of things to note… Firstly, this was not a bug in the glibc package. Secondly, I fixed the bugs in the pacman conflict checking code. So… just maybe… I am not as “incompetent” as the emails I received claimed and I should not be “forced to stand down” as an Arch Linux developer.

Finally, if you are going to send angry emails to me, at least attempt to make them well enough worded that they can join my “angry email hall of fame”. The latest batch were surprisingly uncreative… In fact, the two emails claiming I would be responsible for the authors failing a course due to them spending time fixing their system instead of doing work were just stupid. Why would you update right before a major assignment is due and why would you send me an email if you are running late? Also, the style of the emails makes me think this was one person sending from two different accounts, so they seem to have plenty of spare time… I did enjoy a comment in the bug tracker that said the glibc maintainer was going postal.

My earlier post about installing Arch Linux on a MacBook Pro 5.5 is one of the most accessed posts on my site, so I figured I should write an update for the newer model.

The basic specs of my MacBook Pro 8.1 (13″) are:

• Intel Core i7-2620M @ 2.7 GHz
• 8GB (2x4GB) 1333MHz DDR3 SDRAM
• 750GB SATA @ 5400 RPM
• Intel HD Graphics 3000
• Broadcom BCM4331 802.11a/b/h/n

Installation: I have gone for a pure x86_64 install this time. The initial install was “fun”… So much so, that I would have probably abandoned Arch altogether if I did not have a vested interest in it. The latest official Arch Linux install CD (2011.08.19) does not even boot so I had to grab a testing iso. The install is fairly routine as far as a single OS install on a MacBook Pro goes. I followed the same strategy as my previous install and changed the partition table format and blessed the partition /boot was on for a faster boot-up. I could have tried GRUB2 with its EFI support, but I just stuck with what I knew worked. But “worked” is a funny term as the current Arch Linux installer will only allow you to install GRUB on the hard-drives MBR and not onto an individual partiation (which is required on MacBook Pros). So I did my first manual GRUB installation and everything booted fine!

Video: Just pacman -S xf86-video-intel and everything works.

Screen Brightness: Worked out of the box.

Keyboard Backlight: I recommend learning to touch type… but I read it works.

Touchpad: Sort of worked out of the box using xf86-input-synaptics-1.5.99.902 (see my previous post about what I consider “bugs” in synaptics finger distance calculations). That includes two/three finger right/middle clicks and two finger scrolling. Also, click with thumb and drag with finger no longer required a patched kernel module.

Wifi: Requires b43-firmware from the AUR.

Suspend to RAM: I set xfce4-power-manager to suspend on lid close it worked fine.

Webcam: Worked out of the box.

Sound: Use alsamixer to unmute the speakers.

Keyboard: Screen brightness keys worked fine. Needed to add shortcuts to XFCE for volume control and disc eject.

Fan: Appears fine out of the box, but I need to test it under more variety of load.

Anything else (bluetooth, thunderbolt) has not been tested because “meh”.

Overall, this install took me far less time to do compared to installing on the 5.5 model. There were no patched modules for the touchpad or screen brightness control and no compiling a proprietary module for the wireless. In fact, after the initial dramas with the installation media, everything basically just worked. I guess some of the reason for that is that the 8.1 model I am using was released some time early in 2011, so many of the issues people may have faced early on appear to have already been fixed.