Given the number of unpleasant emails I have received over the last few days, it appears that people have forgotten what the purpose of the [testing] repo in Arch Linux is for. Wait for it…. testing packages! Who would have thought!
Now to clarify some things. While the [testing] repo is for testing, we do our best not to break things in it. In fact, the [staging] repo was set up to allow rebuilds to occur away from the main repos and allow [testing] to always be in a usable state. Also, many (maybe even most) of the Arch developers use the [testing] repo on their main computer. I know I use it on all of mine, including on my work laptop. So I really do not want things to break either. And of course, if I cause breakage, that limits my ability to yell at other people who break packages.
One of the common criticisms I got in my “fan mail” was that I did not do enough testing of the change that saw the /lib directory change into a symlink to /usr/lib. Perhaps the need for testing was why I put the package in the [testing] repo… But, also I did do a fair amount of testing of this update. I tested that pacman stopped the update when there was a file in /lib (or in a subdirectory of /lib), both when the file was owned or unowned by a package. I checked that pacman -Syu --ignore glibc && pacman -Su worked from and old base install.
What I did not test was using pacman -Sf glibc to resolve a conflict, mainly because even with my low expectations on the general population’s intelligence, I did not expect Arch Linux [testing] users to be quite that incompetent. I also did not test having another package owning the /lib directory or a subdirectory within that folder but with no actual files in it (or having its files manually moved from in it). And I did not test upgrading from a more complete install that could have packages with versioned dependencies on glibc (which does not actually break the update but makes it a bit more difficult).
Now ignoring the usage of -Sf… the only case I did not test that actually causes breakage was an empty subdirectory in /lib or another package owning /lib with no actual files in there. That should never actually happen, but it appears people manually moved module or firmware files from /lib to /usr/lib without fixing thier packages, creating this situation. In both these cases, pacman would get to extracting the /lib symlink, see there was still a directory there and refuse to extract. That left people with a system looking for the linker in /lib, but it not being there. Annoying, but this situation is easily recoverable, even without a rescue CD.
You may ask why pacman did not detect those conflicts before attempting the extraction of the package. That would be a good question… It seems the situation of changing a directory to a file or symlink is not a very common operation and so was not very well tested in the (quite extensive) pacman test-suite. Also, a bug fix in pacman 4.0 (prevent removal of empty directories if they are owned by another package), will have caused this bug to be exposed much more frequently. A couple of patches to pacman and these conflicts are now caught prior to the upgrade taking place. These patches are now applied to our pacman package, so non-testing users will not be exposed to this issue.
So a couple of things to note… Firstly, this was not a bug in the glibc package. Secondly, I fixed the bugs in the pacman conflict checking code. So… just maybe… I am not as “incompetent” as the emails I received claimed and I should not be “forced to stand down” as an Arch Linux developer.
Finally, if you are going to send angry emails to me, at least attempt to make them well enough worded that they can join my “angry email hall of fame”. The latest batch were surprisingly uncreative… In fact, the two emails claiming I would be responsible for the authors failing a course due to them spending time fixing their system instead of doing work were just stupid. Why would you update right before a major assignment is due and why would you send me an email if you are running late? Also, the style of the emails makes me think this was one person sending from two different accounts, so they seem to have plenty of spare time… I did enjoy a comment in the bug tracker that said the glibc maintainer was going postal.
Ignore those idiots. Your work and the work of the other arch devs is very much appreciated.
Quite a few ArchLinux users are ex-Debian where testing is for using and stable is apparently more stable than the stable testing. Perhaps peeps got confused.
Archers are known for being quite forthright (read:offensive) sometimes; which is a shame, but having a pop at a volunteer is quite literally self-defeating. You’d have to be pretty stupid to bite the hand that feeds, especially when Google gives a solution quite easily.
best.
A new-ish Archer.
Please name the spammers, at least we can recognise them when they bitch on the forum 🙂 I guess the intersection between them and those of “when will XXX be ready/in extra/packaged” is far from being the empty set.
And, by the way, if you need some words of appreciation, here they are: “thanks for the great free work of packaging all such head-aching stuff”.
Cheers 🙂
I can’t believe some people email you these kinds of things. IMO, you are an integral part of what makes Arch Linux great.
in the archlinux website we can read about testing repo:
“Be careful when enabling [testing]. Your system may break after you perform an update with the [testing] repository enabled. Only experienced users who know how to deal with potential system breakage should use it.”
“Warning: You are expected to be reading the arch-dev-public mailing list if you use the testing repositories. You should know how to downgrade packages and chroot into your Arch install to fix breakage.”
So… we have our responsability if we decide to active it 😉
When I decide to activate it was to help if possible in testing software and to learn about my linux system.. and I learn a lot.. so thanks for your job..
It seems we have a fundamental problem, in that the a class of edge cases (let’s call it “people having done stupid shit”) is not being tested well enough. The reason being that the early testers tend to know what they are doing and hence not have done too much stupid shit on their machines. We try of course to imagine what sorts of stupid shit people might have done, but it seems we keep underestimating the ingenuity of idiots.
On the bright side, this sort of thing is very educational. People will learn more about how pacman and packaging works, and I personally found it interesting to see the neat ways of solving this problem without a rescue disk.
By the way, I’m a bit miffed that I didn’t get to share any of the blame for this. Didn’t get a single piece of hate-mail… 🙁
but it seems we keep underestimating the ingenuity of idiots.
That’s the only breakage I am aware of…
I can help with not getting any mail about this problem with some about another one if you feel left out.
Myra
Regarding Debian testing, I believe they have set the confusing terminology.
In Arch linux, I don’t have time to fix the os all the time, what with a full-time job and all. But hey, that’s why I don’t use testing on any of my PCs. When I do programming at my job, which is semi-production, I have a *testing* server and a *stable* server for the same reason. Programmers/devs/packagers need a place to post where people can test the latest before going mainstream. No flames allowed in testing!
But seriously, thanks for all your work. I’ve been an Arch linux user for years, and I even submitted a few AUR packages, but never touched testing. IMHO Arch is the best OS for learning linux while using it regularly.
Actually, [testing] is pretty stable in my experience. It’s so stable that I had to go backback and read some of the documentation to figure out to fix my system since it had been a while. And of course, people will sooner send you email when they have problems than when everything went smoothly.
Wow. lol some people are daft to say the least.
Thanks for the good work and funny blog.
I think you should ignore such people. I understand it can be hard. Jason’s recent post (http://jasonwryan.com/blog/2012/06/16/misunderstanding/) on the planet right highlights the issues of how the contributors, such as yourself, and the contributions are often misunderstood.
We can only hope, that everybody in the Arch community understands that.
Thanks for all your work.
Cheers.
To say the least, I think you did a great job with the update. As Tom said
“it seems we have a fundamental problem, in that the a class of edge cases (let’s call it “people having done stupid shit”) is not being tested well enough. The reason being that the early testers tend to know what they are doing and hence not have done too much stupid shit on their machines. We try of course to imagine what sorts of stupid shit people might have done, but it seems we keep underestimating the ingenuity of idiots.”
Now i’m generally considered one of those idiots, something I can’t help I’m miswired. It’s also the reason I use the testing repos — it helps me learn. FYI I got this one right because I’ve learned to follow arch-projects, arch-general, dev-public, and read the Arch news.
I think you and all the devs are doing a fantasic job. Keep up the good work and thanks to you and all the arch devs for devoting their time and effort to make Arch Linux a fantastic distro.
Keep up the good work. Ignore those angry silly mails. Thx.
Well, these people are obviously idiots. There was information on how to resolve this issue on reddit and the Arch Linux forums and mailing list within hours of the problem appearing. This is in addition to the information posted *before* the new glibc package was pushed to [testing] explaining the whole /lib symlink situation. You can only do so much…
Thanks for all your work.
I think you should offer all the hate-mailers a full refund.
Thanks for brining me a painless and well working upgrade of /lib. I am a bit confused without /lib, /lib has always been there but I guess I will have to learn and improve 😉
I very much liked the way you posted information of the update on the home page, which I follow by RSS to know those kind off attended updates in fore hand.
I also like the effort you put in articles describing possible failures along the way. I only missed if I had to reboot so I did, just in case 😉
Upgrades on Arch has, during the months or half a year I have tested it, been smooth and painless, the main reason I moved from Gentoo.
All in all, thanks for the good work you and other maintainers bring and that make my systems tick!
People are morons, it happens even in “the land of Arch” (nice title for the fantasy book). Keep rocking, you’re all doing great work!