Comparison of Security Issue Handling

More follow-up from the afore mentioned Frostcast featuring Manjaro developer Philip Müller. Just past the 16 minute mark.

We learn and everyone makes mistakes. And the new server change every package is new synced from Arch Linux so there is no security issues. … We sync daily so if there is any problems with our system it’s ninety percent from Arch itself, so I don’t know why they bash us.

I am not going to claim Arch is the bastion of all things security – in fact I know Arch is far from perfect here – but Manjaro claiming that they are on par with Arch is wrong. Saying “we sync daily” is frankly deceptive. The daily syncs are to the Manjaro unstable branch, so packages can take a while to reach the stable branch where the vast majority of users get the package. As I have pointed out previously, Arch does not separate out security updates from plain upstream updates, so when Manjaro holds back updates on the unstable branch in the name of stability, they are also holding back security fixes. The updates need monitored for security fixes and either 1) pushed more quickly to the users, or 2) have the fixes backported to the “stable” packages.

But, lets use an example, because facts are good. Recently there was an privilege escalation issue found in polkit. This was made public on 2013-09-18. And over the next couple of days there were a lot of distribution updates to fix this issue. So, I have not picked an obscure bug given the number of distros dealing with the issue, and it is a privilege escalation one (potentially with proof-of-concept available, although I have not checked that out). Lets compare the Arch and Manjaro response to this issue by monitoring the location of the polkit-0.112 package:

Date Arch Manjaro
2013-09-18 Testing -
2013-09-19 Stable -
2013-09-20 Stable Unstable
2013-09-21 Stable Unstable
2013-09-22 Stable Unstable
2013-09-23 Stable Unstable
2013-09-24 Stable Unstable
2013-09-25 Stable Unstable
2013-09-26 Stable Unstable
2013-09-27 Stable Unstable
2013-09-28 Stable Testing
2013-09-29 Stable Testing
2013-09-30 Stable Testing
2013-10-01 Stable Stable

I will admit that this is actually better than I thought it would be… I thought packages stayed longer in Manjaro’s testing repositories to catch bugs. Then again, I noticed that there are packages that were pulled into Manjaro from Arch and put into their stable repos within ten minutes, including packages in the [core] repo, so I’ll assume that the testing that occurs in the Unstable and Testing branches is rather limited. (Evidence: pool/ directory with timestamp file was synced from Arch, stable/extra/x86_64/ directory with repo database timestamp.)

In summary, the indiscriminate holding back of all updates in the name of testing(?) is why I “bash” Manjaro security. With this system, Manjaro is always running behind Arch, so claiming the Manjaro security issues are “ninety percent from Arch itself” is full of… optimism. And before the “leave Manjaro alone” comments, I will stop posting about it when I have no need to correct such false statements.

Keeping Packages Vanilla – 1. Patching

I have recently been thinking about a Linux distribution providing “vanilla packages” and what this really means. The basic idea is simple – provide packages as upstream released them. In practice, this is an impossibility (for reasons I will cover below).

Firstly, lets cover the reason for providing vanilla packages. And I use “reason” singular deliberately, because I think it comes down to a single point. The software developer knows their software better than you do. Your patch could unintentionally introduce a security issue – it has happened before… Or you could introduce a new feature that is then introduced by the developer in a different and incompatible way in the next official release. Also, any bugs that are found in your modified piece of software will need to be triaged in an unpatched version of the software in order to report the issue upstream.

So, in an ideal world, we would just run the equivalent of “./configure; make; make install” and all software would install perfectly. But the world is far from ideal… I will cover two points across two posts: firstly patching, followed by configure options and dependencies.

Patching software is a necessity in any Linux distribution. I am only considering rolling release distributions in the discussion below, so that removes backporting fixes and features from newer versions of a software package. So, what patching is minimally required:

  1. Patches for build issues
  2. Patches for security issues
  3. Patches to fix major software features

I am completely ignoring patches released by upstream as part of their update process. For example the Linux kernel provides a large patch that updates from their x.y.0 release to x.y.z. Bash releases a patch for each minor update, so bash-4.2.042 requires applying 42 patches… These are obviously required patching and are fully sanctioned by the developer so do not deviate from vanilla.

It should be fairly obvious what patches for build issues are… the software will not compile for some reason so you need to do patching to fix that. A piece of software is almost never tested in every single environment before release and, even if it was, updates to other pieces of software can cause build issues. This is particularly common with gcc updates which have become progressively stricter on which headers needed included for a function. Even worse, a software developer might release with the “-Werror” flag enabled by default, meaning any new warning will result in a build failure (I do not have kind words about software developers that do that…). Then there are more complicated issues involving a library update with API changes requiring much more extensive fixes. While adding a single extra include really does not require upstream approval before applying, even that should be forwarded upstream.

Security issues are an important part of patching for non-rolling release distributions. However, new versions of software are usually released whenever a security hole is found, so rolling release distributions only need to update. Again, just grabbing a patch from anywhere on the internet and applying it is not a good idea – I have seen this actually result in a larger security issue than the original.

The final category of patches are those that fix major software features. For example, if an IRC client has a bug preventing it connecting to any channels, firstly shake your fist at the developer and tell them to do some testing before release, and then patch it. If there is a typo in the help output, file a bug or submit a patch upstream, but there is no need to patch it. The guiding principle should be something like “if this is the only issue found in the software, would the developer consider making a new release?”. The answer to that question “yes”, then it will be “yes” to provide the patch.

One guiding factor that can not be stressed enough here is that all patches should be approved by upstream. The best situation is if upstream have committed the patch into the version control repository – preferably on the branch for the version you are using so no mistakes can be made back-porting. Failing that, a post on a mailing list or bug tracker by one of the main developers of the software approving a patch is acceptable.

Of course, much of this is subjective. Is that broken feature big enough to patch? Does this bug constitute a security issue? If upstream is rather unresponsive at the moment, should I apply this fix for a security bug? Is this build fix minor enough that I do not need to wait on an upstream comment before applying? Given it is hard to formulate these ideas into precise rules, I think the answer becomes one of how strict the packager is. I was far more likely to include patches when I started packaging for Arch Linux than I am now. So maybe it is not how strict the packager is, but rather how grumpy…

Manjaro Linux: A Follow-Up

And… do you know how easy it was to convince a Manjaro developer to ignore their entire stability scheme and push a known broken package into their repos?

The firefox-18-1 package in Arch Linux was known to have an issue with language packs. So all non-English users would have their interface changed into English. The behaviour was the same for dictionaries.

Users were complaining about this issue on the Arch Linux mailing list and forums. But make a comment about it and the entire stability mantra of Manjaro Linux is ignored and the package can is moved into the Manjaro stable repositories (that is about 8 hours after my posting).

So even known usability issues with packages, which is now fixed in the Arch Linux package firefox-18-2, do not get caught by Manjaro’s stability testing. I do not think I have any need to feel threatened

Edit: It appears I misunderstood Phil’s comment in the last post exclaiming “Don’t know what the fuss is all about ?!?” with link to packages moving between repos to mean he moved it to the stable repository. It turns out it only moved to their testing repository. So, users are still exposed to security issues, which is what the fuss was about. And it is only the move from “unstable” to “testing” that showed a lack of the claimed quality control. So apologies to Manjaro – I only showed you have packages with security issues and the stabilization stage controlled by your developers is worthless.

Manjaro Linux: Ignoring Security For Stability

I feel like having a rant today… so nothing particularly unusual there. But after reading yet another post saying:

I used Arch for two years and it was perfectly fine until one day when I updated and it broke my system. Now I have been using Manjaro for a month and it is completely stable.

I have found what to rant about! Is it just me that notices the issue with that statement?

I have no issue ranting about Manjaro because every time I read their forums I see one of their Core Team being less than congenial regarding Arch – but I suppose they have to be given one of the main selling points of their distribution is they can fix the Arch Linux updating “mess”. Also, the defensiveness of their community to any criticism and the amount of self congratulations for being a Manjaro user astounds me – and I spend a lot of time in the Arch forums and IRC channel where the community are widely considered to be elitist pricks (with me being no exception, as this post will plainly show).

For those who do not know how Manjaro works, I will paraphrase this post. The Arch stable repos are synced into Manjaro Unstable on a roughly daily basis. They sit there for 1-2 weeks before being declared stable and moving to Manjaro Testing. Then their test squad declares that stable enough to move to Manjaro Stable, about 3-4 weeks after the packages arrive in Arch Linux.

And this is the issue. There is four weeks until Manjaro users get package updates. That is still a lot quicker than a non-rolling release distribution I hear you say, but it ignores one of the fundamentals of a rolling release distribution. Security fixes come with a new software release. On a fixed-point release distribution, security fixes are backported into your out-of-date software versions to maintain stability. On a rolling release distribution, you just release the newer version of the software that comes with most security fixes (some backporting from the upstream VCS is required if a release is not made).

That means, Manjaro users are vulnerable to security bugs for around a month after Arch users are safe, unless of course the Manjaro Core Team monitors every package and pushes those with security fixes. How many packages in a distribution? Arch Linux has >6000 in its binary repositories. I suppose it is not impossible to monitor that many packages, unless of course your Core Team consists of three people. And given those three people provide five variants of their installation ISO (net install, XFCE, KDE, Cinnamon, MATE – with OpenBox and E17 on the way…) and provide a series of kernel packages and systemd… Things are looking bleak.

And so, Manjaro users are stuck with packages having security issues for a while. I’d assume the big ones get through quicker. Although their firefox package has not been updated to version 18 yet, which fixes 21 security issues – 12 of which are marked critical. In fact, firefox version 18 has not even made their Unstable repo as I am writing this…

Lets say Manjaro had the man-power to monitor all updates in Arch Linux for security issues. Could they be brought to the Stable repositories more quickly? Maybe… But remember Arch Linux rebuilds against new versions of libraries with soname bumps all the time and our toolchain gets updated very quickly after any upstream release. So any security update built against new libraries or with a new toolchain version require those components moved too. And they are the types of updates that could introduce stability issues.

In the end, I think the idea behind Manjaro – rolling release at a more relaxed pace – can be achieved. I am not entirely familiar with these distributions, but I guess that is exactly what apotsid and LMDE achieve. And they start form Debian Unstable, which is reportedly far more of a minefield than Arch Linux.

Ubuntu is an Evil Dictatorship!

It seems that the Ubuntu community has found out that their opinion does not count as much as they thought and that Ubuntu “is not a democracy“. So the purple, I mean aubergine, theme is here to stay and the window control button placement will be on the top left (and in the opposite order from OSX for the moment). I guess this is a really big issue because you can not change themes or configuration files in Ubuntu… wait… you can? Oh well then, move along, nothing to see here. I can almost guarantee that there will be a script released that changes the window control placement to the right side, just as many scripts are available to install all the “restricted” multimedia codecs that are not installed by default.

We all have known for a long time that Arch is not a democracy. So Ubuntu users set to move to a new distro where they can contribute nothing but still have their opinion count should not look towards Arch.