How Secure Is The Source Code?

With the addition of source code PGP signature checking to makepkg, I have began noticing just how many projects release their source code without any form of verification available. Or even if some form of verification is provided, it is done in a way that absolutely fails (e.g. llvm which was signed by a key that was not even on any keyservers meaning it could not be verified). If code security fails at this point, actually signing packages and databases at a distribution end-point instills a bit of a false sense of security.

To assess how readily validated upstream source code is, I did a survey of what I would consider the “core” part of any Linux distribution. For me, that basically means the packages required to build a fairly minimal booting system. This is essentially the package list from Linux From Scratch with a few additions that I see as needed…

For each source tarball I asked the following questions: 1) Is a PGP signature available and is the key used for signing readily verified? 2) Are checksum(s) for the source available and if they are only found on the same server as the source tarball, are they PGP signed? The packages are rated as: green – good source verification; yellow – verification available but with concerns; red – no verification. Apologies to any colour-blind readers, but the text should make it clear which category each package is in…

Package Verification
autoconf-2.68 PGP signature, key ID in release announcement, key readily verifiable.
automake-1.11.3 PGP signature, key used to sign release announcement.
bash-4.2.020 PGP signature for release and all patches, link to externally hosted key from software website.
binutils-2.22 PGP signature, key used to sign release announcement (containing md5sums).
bison-5.2 PGP signature, key ID in release announcement, externally hosted keyring provided to verify key.
bzip2-1.0.6 MD5 checksum provided on same site as download.
cloog-0.17.0 MD5 and SHA1 checksums in release announcement posted on off-site list.
coreutils-8.15 PGP signature, key used to sign release announcement.
diffutils-3.2 PGP signature, key used to sign release announcement.
e2fsprogs-1.42.1 PGP signature, key readily verifiable.
fakeroot-1.18.2 MD5, SHA1 and SHA256 checksums provided in PGP signed file, key readily verifiable.
file-5.11 No verification available.
findutils-4.4.2 PGP signature, link to externally hosted key in release announcement.
flex-2.5.35 No verification available.
gawk-4.0.0 PGP signature, key difficult to verify.
gcc-4.6.3 MD5 and SHA1 checksums provided in release email. MD5 checksum provided on same site as download.
gdbm-1.10 PGP signature, key ID in release announcement (with MD5 and SHA1 checksums), key readily verifiable.
gettext-0.18.1.1 PGP signature, key readily verifiable.
glibc-2.15 No release tarball, download from git (PGP signature available when release tarball is made).
gmp-5.0.4 PGP signature, key ID and SHA1 and SHA256 checksums on same site as source, key difficult to verify otherwise.
grep-2.11 PGP signature, key used to sign release announcement.
groff-1.21 PGP signature, key difficult to verify.
grub-1.99 PGP signature, key used to sign release announcement.
gzip-1.4 PGP signature, key used to sign release announcement.
iana-etc-2.30 No verification available.
inetutils-1.9.1 PGP signature, key readily verifiable.
iproute-3.2.0 PGP signature, key readily verifiable.
isl-0.09 No verification available.
kbd-1.15.3 File size available in file in same folder as source.
kmod-0.05 PGP signature, key readily verifiable.
less-444 PGP signature, key posted on same site as download, key difficult to verify otherwise.
libarchive-3.0.3 No verification available.
libtool-2.4.2 PGP signature, key readily verifiable, MD5 and SHA1 checksums in release email.
linux-3.2.8 PGP signature, key readily verifiable.
m4-1.4.16 PGP signature, key used to sign release announcement.
make-3.82 PGP signature, key used to sign release announcement.
man-db-2.6.1 PGP signature, key used to sign release announcement.
man-pages-3.35 PGP signature, key readily verifiable.
mpc-0.9 (libmpc) PGP signature, key readily verifiable.
mpfr-3.1.0 PGP signature, key readily verifiable.
ncurses-5.9 PGP signature, key used to sign release announcement.
openssl-1.0.0g PGP signature, key readily verifiable.
pacman-4.0.2 PGP signature, key readily verifiable.
patch-2.6.1 PGP signature, key difficult to verify.
pcre-8.30 PGP signature, key readily verifiable.
perl-5.14.2 MD5, SHA1, SHA256 checksums provided on same site as download.
pkg-config-0.26 No verification available.
ppl-0.12 PGP signature, key readily verifiable.
procps-3.2.8 No verification available.
psmisc-22.16 No verification available.
readline-6.2.002 PGP signature for release and all patches, link to externally hosted key from software website.
sed-4.2.1 PGP signature, key difficult to verify.
shadow-4.1.5 PGP signature, key readily verifiable.
sudo-1.8.4p4 PGP signature, key difficult to verify.
sysvinit-2.88 PGP signature, key difficult to verify.
tar-1.26 PGP signature, key used to sign release announcement.
texinfo-4.13a PGP signature, key difficult to verify.
tzdata-2012b Many checksums provided in release announcement.
udev-181 PGP signature, key readily verifiable.
util-linux-2.21 PGP signature, key readily verifiable.
which-2.20 No verification available.
xz-5.0.3 PGP signature, key difficult to verify.
zlib-1.2.6 MD5 checksum provided on same site as download (although download mirrors available).

Note that some of these packages have additional methods of verification available (e.g. those that are PGP signed may also provide checksums and file sizes), but I stopped looking once I found suitable verification. When I label a key as “readily verifiable”, that means it is either signed by keys I trust, that it is used to sign emails that I can find or it is posted on the developers personal website (which must be different from where the source code is hosted). I personally found my preferred method of verification was packages whose release announcements were signed by the same key as the source.

While you might look at that table and think there is a lot of green (and yellow) there so everything is in reasonable shape, it is important to note that the majority of these are GNU software and all GNU software is signed. Also, 15% of the packages in that list have no source verification at all. From some limited checking, it appears the situation quickly becomes worse as you move further away from this core subset of packages needed for a fairly standard Linux system, but I have not collected actual numbers to back that up yet.

Anime Guide 2011

Last year’s anime list appeared in July, so I am definitely making progress at getting this done earlier in the year. Who knows, by next year it might appear in January! Then again, it could happen in August…

As always, this is not a review of all of the year’s anime as there is far too much crap out there. Instead it is a collection of my opinions on the anime that I thought were good enough to watch in the first place. There was definitely a lot of decidedly average anime during this year. There was not necessarily anything bad about them, just there is nothing to make them stand out as worthwhile either.

Anime of the Year

Steins;Gate

Steins;Gate
(TV, 25 episodes)

It took me until at least the eighth episode to have some sort of idea what was going on here. Bananas in microwaves that turn into green goop… But even though I had no idea what was happening, the early episodes still are intriguing enough that you want to watch more and it gets even better once the plot points all begin coming together.

Recommended

Black Lagoon - Roberta's Blood Trail

Black Lagoon – Roberta’s Blood Trail
(OVA, 5 episodes)

If you like the two previous series of Black Lagoon, then you will probably like these episodes. It is just more of the same thing, but given the previous series were very, very good…

Mawaru Penguin Drum

Mawaru Penguin Drum
(TV, 24 episodes)

This series made it into my Recommended list based entirely on its uniqueness. And I am sure there is some deep underlying meaning in there (probably something about abandoned children being able to find happiness) and there appears to be a large number of metaphors too (the boxes, the apple). But there were too many twists and turns for me to fully grasp what was being implied. I still have no idea what the penguins were…

Tiger and Bunny

Tiger and Bunny
(TV, 25 episodes)

Remember those days of getting up early on a Saturday morning to watch cartoons? Tiger and Bunny is like an American superhero cartoon, but with more awesome. Almost enough awesome for me to select it as the anime of the year. I even found the shameless sponsor promotion to be quite amusing. But in the end, there is nothing amazing or original about this series, it is just damn entertaining.

Usagi Drop

Usagi Drop
(TV, 11 episodes)

Not the usual style of show for me to watch, but it was a solid drama so the risk paid off. However, if you like this show and want to find out more by reading the manga, then you should wipe that thought completely from your mind. Pretend the manga never existed. Seriously… No arguing. Just don’t. As one blogger wrote, he should have just let sleeping, scrotum-eating, rabid dogs lie.

Average

A Certain Magical Index II

A Certain Magical Index II
(TV, 24 episodes)

Worse than the first series and far worse than Railgun. But given both of those were very good, that is not bad in itself. However, my overall impression was that they were running out of ideas so they had to stretch out the little that they had, which resulted in a fairly slow pace. And the superpower that only negates other superpowers gets boring.

Blue Exorcistt

Blue Exorcist
(TV, 25 episodes)

There was a few quite good episodes in this series. However, its storyline is overly predictable and has too much filler (including a beach episode…).

Broken Blade

Broken Blade
(OVA, 6 episodes)

I have never been a fan of mecha anime, but this was one of the better ones. Well, it was for the first four episodes anyway. I found the finish was bland. Almost as bland as the characters… But at least they were not teenagers because the addition of boring teenage coming of age crap (that normally is in mecha anime), would have killed this for me.

Deadman Wonderland

Deadman Wonderland
(TV, 12 episodes)

The theme park style prison where inmates have to compete in fights in order earn points to buy their continued living is not particularly original… But at least the superpowers were interesting. The ending is also far too non-conclusive, but I am not sure if it was because there was nothing to conclude or if everything was kept open to make a sequel.

Dragon Ball Kai

Dragon Ball Kai
(TV, 97 episodes)

I watched this mainly for nostalgia reasons and because I have never seen the final 50 or so episodes from Dragon Ball Z. While this version was significantly more fast paced than the original (the 97 episodes covered what took 193 in DBZ), I think it could still have moved faster in lots of places. Also, the remake is not complete and so I still have not seen the second half of the Majin Buu Saga. Maybe one day…

Fractale

Fractale
(TV, 11 episodes)

This anime started off well, but got progressively worse. But I can not actually say how it got worse… it just became bad. And I think I mean genuinely bad and not just bad because it did not live up to the potential shown in the first episode.

Gosick

Gosick
(TV, 24 episodes)

This tries so hard to be a good show. But I think I am already bored of shows that involve some sort of detective with a side-kick. The first few cases are examples of entirely amateur writing. I think someone just jumped on the internet and searched for “crime cases with minor twist when solving”. It all just seemed too familiar. But the series was rescued as story did go beyond a pure detective drama and turn into something much more interesting.

No. 6

No. 6
(TV, 11 episodes)

After lots of war, only six pieces of land remain. Guess which number this is? This is another series that I think started off strong but just did not go anywhere spectacular. Just a typical Dystopian Society storyline.

Subpar

The Mystic Archives of Dantalian

The Mystic Archives of Dantalian
(TV, 12 episodes)

You can always tell a show is bad when there is some sort of super power transformation sequence that lasts so long that you starting thinking to yourself that you could have killed both of the lead characters multiple times by now and that would not necessarily be a detriment to the show. It is even worse when that superpower is unlocking a girls chest with a key and then reaching in a pulling out a book. That takes forever, but then the book has to be read aloud for it to work. With only minor exaggeration, in total this sequence takes about three quarters of every episode.

Ten Years of Arch Linux

Today marks the 10 year anniversary since the first release of Arch Linux. I have been involved in Arch for only about half that time, but I thought it might be quite interesting to make a time-line of the major things I remember being involved with in my history with Arch Linux.

I actually first used Arch Linux in a virtual machine for quite some time prior to the start of this time-line. I was running a Linux From Scratch install and was looking to add better package management. Then I realized that it was quite a waste of time building my own packages when Arch was actually quite similar to what I was trying to achieve in a Linux install. The toolchain was the part I liked dealing with the most in LFS and now I get to play with it anyway, so things worked out well!

Here is a brief summary of my involvement with Arch Linux:

2007-06-09 – Joined forums
2007-06-20 – First non-VM Arch install
2007-12-04 – First patch to the pacman project (fixed compressing man pages in makepkg)
2008-01-09 – Applied to be a Trusted User (TU)
2008-01-23 – Voted in as a TU (23 yes, 1 abstain!)
2008-06-03 – Joined the forum moderation team.
2008-06-04 – Became a Arch Linux developer!
2008-08-17 – First big rebuild. I made all ncurses applications use the wide-character library for better locale support.
2008-09-01 – Interviewed in the Arch Newsletter – yes, there used to be one of those…
2009-04-04 – Resigned as a TU
2009-04-22 – First complete toolchain build after becoming the toolchain maintainer (gcc-4.4 update)
2009-06-07 – Broke module-init-tools due to missing a file failing to compile in the PKGBUILD. (makepkg would catch such an error these days and the missing file would be noted using the checkpkg utility)
2009-07-17libjpeg-7 rebuilds enter [extra]. I think this was the first major rebuild to severely break KDEmod.
2009-08-02pacman-3.3.0 released with package splitting support in makepkg
2010-03-05 – Fix a bug introduced by my first patch to makepkg… In true Allan style, the commit message contains a typo so does not point at the right bug!
2010-07-02 – Resigned as forum moderator. Mainly because I wanted to be able to be grumpy on the forums when…
2010-10-18python3 transition – Probably the single most controversial thing I have done in Arch…
2011-02-28 – A minor bash update resulted in non-booting systems
2011-03-16pacman-3.5.0 released for which I contributed reading the sync databases directly from the downloaded tarball, disk space checking and storing package information in a hash table.
2011-10-13pacman-4.0.0 release with full PGP signing support for packages and databases.

Of course there have been lots of other things I have contributed to Arch, including 1000s of package updates and about 400 patches to pacman. But these are the ones I found most memorable.

Windows Service Center

I just had an interesting phone call from someone at the “Windows Service Center”. Note I spelled “Center” the non-Australian way, because I do not think they were based here. Anyway, I get these calls about once a month so I get to have a bit of fun. The conversation went like this…

Caller: “Hello. Is Mr. Mac-Rae there?” (my name was pronounced with a discernible gap…)
Me: “Yes”
Caller: “(pause…) Are you Mr Mac-Rae?”
Me: “Yes”
Caller: “I am calling from the Windows Service Center.”
Me: “My windows are quite clean from all the rain we had lately and they are showing no leaks.”
Caller: “Um… as in Windows on your computer.”
Me: “Oh, right. That would make lots more sense.”
Caller: “Are you the main user of the computer?”
Me: “Which one?”
Caller: “You have more than one?”
Me: “Yes. Six” (which is technically correct – the best kind of correct – but most of them are broken and serve as door stops)
Caller: “We will start with the one you mainly use.”
Me: “OK then. Yes, I am the main user of the computer I mainly use.”
Caller: “There are lots of viruses on the internet these days and our records indicate that your computer is infected.”
Me: “Which one?”
Caller: “Excuse me?”
Me: “Which computer is infected?”
Caller: “Probably all of them, but I will help you check.”
Me: “How would I know such a thing?”
Caller: “Have you ever seen a ‘This page can not be displayed’ message in Internet Explorer?”
Me: “I can honestly say I have never seen that message from Internet Explorer on my computer.”
Caller: “OK. We better check your computer anyway. Can you go to the Control Panel?”
Me: “I don’t think my computer has one of those.” (XFCE calls it a “Settings Manager”)
Caller: “I will guide you to it. Is your computer on?”
Me: “No. I just turned it off because I was told it might be infected with a virus.”
Caller: “Can you please turn it back on?”
Me: “OK, but it will take a while. It seems to be booting really slowly lately for some reason…”
Caller: “OK, I will wait.”

(a few minutes interlude while I respond to an email)

Me: “OK, my computer is on.”
Caller: “Good. Now click on the button in the lower left corner of the screen.”
Me: “I do not have a button there.” (slight lie… I have a “Show Desktop” button there)
Caller: “What operating system do you use?”
Me:“Plan 9”
Caller: “Sorry, what was that?”
Me:“Plan 9”
Caller: “…” (hung up)

So I probably went a bit overboard with choosing Plan 9 as the operating system I was running, but I was getting bored. One day I really should follow instructions and see what they try to get me to do to my computer.

International Rare Disease Day

I normally do not put much personal information on this blog, but given today is International Rare Disease Day and I have experience with a very rare disease, I thought I would make an exception. In fact, this disease is so rare, it often appears in lists of the top ten rarest diseases (although my suspicion is that those lists are crap… but it is still extremely rare).

My issues first started in July 2010 when I began getting ulcerations on my tongue and cheeks. I initially thought that these were caused by biting in my sleep and figured that it would go away eventually. But it did not, and when I ended up spitting out more blood than toothpaste when brushing my teeth I decided it was time to see a doctor. The doctor I saw also thought it was bruxism and sent me to a dentist. The dentist decided that my wisdom teeth were a probable cause given where the damage in my cheeks was and that they were a bit out of line. So I had them pulled out and was put on various antibiotics to resolve any potential infection that could be involved. When I went back to see the dentist a couple of weeks after the second set of wisdom teeth were removed, he took a look in my mouth and decided there was something else going on.

I then was referred to an oral and maxillofacial surgeon. She took one look at my mouth and then got her receptionist to call an oral pathologist and get me an urgent appointment. Urgent, not because she though it was going to kill me, but because it looked quite bad (basically a direct quote…). She also sent me to get blood tests done to rule out a wide variety of infectious diseases. I did find it kind of amusing that she actually asked me if it was fine to test for some STDs rather than just saying this is what you are being tested for… Anyway, unsurprisingly I was all clear on that front.

Then I was onto seeing the oral pathologist. He spent a lot of time holding out my lip and staring at it. By this time I was starting to get blisters on the inside of my bottom lip and on the roof of my mouth. On the first visit he took two hole-punch biopsies from my mouth – and they are what they sound like… Your mouth gets numbed and a “small” cylinder of tissue is removed. This is also when I started on prednisolone (but more on that later…). Those biopsies came back suggesting I had an autoimmune disorder called lichen planus. The level of prednisolone I was taking was increased but this made little difference. I am fairly certain that the oral pathologist was not convinced of the diagnosis because he did much more lip staring and decided more biopsies were needed. Those came back as general ulcerated tissue (i.e. completely non-diagnostic) and so he sent me back to the oral surgeon so she could take a larger sample. Again, this was non-diagnostic as the tissue was just too damaged to get a clear view of which layer of the dermis the blistering was occurring in. By this stage, I had started getting blistering on my skin so I was sent to a dermatologist.

The dermatologist took a look at me and decided it was most likely a form of oral lichen planus that I had. But to make sure, more biopsies were taken from my lip… As the prednisolone was not helping a lot, it was time to get me onto some immune suppressing medication. That sort of medication is quite expensive so I was brought into the public hospital system where the doctor could apply to get the medication at a greatly subsidized rate. I was started on cyclosporin in February 2011. A biopsy from my finger (local anesthetic to the hand is really painful!) and some antibody tests appeared to confirm the diagnosis of lichen planus. Some antibody levels that would indicate another auto-immune disease affecting the skin were slightly raised, but not enough to change the diagnosis as these levels would not be unexpected given all the damage I had to the skin.

Around this time I started noticing I was getting shortness of breath while exercising. As I was told, you are suppose to get a bit puffed while doing exercise… but it was different. The best I could explain it was that I was feeling wheezy. So I was referred to the thoracic department where I had breathing tests and an ECG done. From those results, I was scheduled to have an echocardiogram and a CT scan of my chest.

While this was happening, there was not much progress in the healing of my mouth, so I was put on acitretin – a drug normally used to treat psoriasis, but also shows some evidence of helping lichen planus. And things did start to improve while on both medications.

I went for the echocardiogram and, from what I now know of how a normal one goes, things were not right. The ultrasound person had their supervisor come in and I was scanned while drinking water. A few days later I received a phone call saying the echo has shown a large mass behind my heart (“fist sized”) and I really should not miss the CT scan in the next couple of days. Also they were arranging for me to be admitted to hospital right afterwards so they could arrange further treatment. At this time it was decided I would probably need an immune system to deal with whatever that mass was, and given the acitretin appeared to be working, I was taken off the cyclosporin.

The CT scan did not give a clear diagnosis about what the mass was, although it did show it was not attached to my esophagus, which was good as I knew from various seminars I attended for work that cancer there is very high on the “not a good thing” scale. While I was in hospital I had a bronchoscopy performed with biopsies taken from my lung, which did not provide anything diagnostic to why I was having difficulty breathing. There was some Pneumocystis bacteria seen at levels that were not really a concern. I was brought back in the next week for an endoscopy where they were going to punch a small hole through my esophagus to get a sample from the mass. That was aborted mid-procedure as instead of having just a small hole punched through the esophagus, a tear occurred instead. The doctor doing the endoscopy had not seen anything like that before.

It was decided to just go in and look at the mass directly and, if it seemed a good idea at the time, to then remove it. So I was scheduled for surgery a couple of weeks later, which was late May 2011. During that two week period, I got an extensive flare-up of the blistering of my mouth and what looked like conjunctivitis in my eyes. When I say extensive, it was really extensive… the entire lining of my cheeks and roof of my mouth was gone and my tongue has extensive ulceration. The best way I have of describing it is to think of the blisters you get on the roof of your mouth when you eat a hot slice of pizza. Now extend them to everywhere. I completely lost all skin on my lips and started getting blistering over my face and ears. The blistering on my hands became a lot more extensive and my fingernail beds became swollen and the base of the nails became thin to the point of not being there. I also had an itchy rash covering most of my body. My endocrine/bone doctor actually wrote on my chart that I looked “horrid” and that the issues I was having due to being on prednisolone for the last eight months (low testosterone levels, bone density dropping…) were the least of my problems. I was also starting to lose a lot of weight given I had difficulty eating anything substantial, but luckily I had backup weight so that never became too much of an issue. I spent a few nights in hospital being put on various drips and having tests done to make sure I got to the surgery in a reasonable condition.

When I went to hospital for the surgery to remove the mass, the surgeon took a look at me and became very concerned to the point where I think he was considering delaying. I think the fact that I was not going to get any better without going on immune suppression again and that it would be better if we knew what the mass was before that happened is what convinced him to go ahead (and probably my wife crying had an influence…). The surgery was successful, with the entire mass removed and leaving me with an awesome looking scar (there are 26 staples there). The doctors could not tell what the mass was when they removed it but a week or so later it came back as being Castleman’s disease. That diagnosis in itself was quite good as removal is usually curative. And while that is very rare, I can still do “better”!

It was with this diagnosis that things started to get placed together. When I saw the dermatologist next he took one look at me and admitted me to hospital. I definitely did not have lichen planus… Taking the Castleman’s into account, it was very likely I had paraneoplastic pemphigus. Another couple of biopsies from my hand and more antibody tests and this was confirmed. That antibody that previously had levels slightly elevated above normal was now off the charts so there was no doubt in the diagnosis. This was also confirmed by the eye symptoms I was having which on close inspection showed the surface of the eye eroding, which is common with paraneoplastic pemphigus.

I was going to need some quite extreme immune suppression. The short term treatment is extremely high doses of prednisolone and all its side effects… But before doctors hammer your immune system to the extent needed, you are checked for every infection they can think of. The infectious diseases team came and saw me and asked questions about my lifestyle that I never think I will be asked again! Anyway, I was once again cleared of all infectious disease tested for. The only concern was the bacteria observed during the bronchoscopy as it can cause a type of pneumonia in people with reduced immune systems. So once a treatment for that was sorted out (because I am potentially allergic to the usual medication), I was back on immune suppressants; this time mycophenolate. After a bit more than two weeks in hospital, it was decided that I no longer needed daily testing and I could manage my treatment at home.

I was out of hospital for a couple of weeks and then had my lungs tested again. Since my previously measurement, there was quite a substantial drop in lung function (for example, the FEV1 measure was now around a third of the expected value). Combined with the diagnosis of Castleman’s disease and paraneoplastic pemphigus, this lead to the diagnosis of bronchiolitis obliterans, which takes the entire situation from being awful to life threatening. In bronchiolitis obliterans, the small airways in your lungs become scarred which prevents the flow of air through them (from what I understand, it is mainly the outwards flow that is affected). I describe it as being like having a permanent severe asthma attack, although that might be entirely incorrect medically.

So it was back to hospital for another two weeks while the best course of treatment was decided. Bronchiolitis obliterans is irreversible, so the best that could be hoped for was stopping the decline in lung function. The immune suppression I was already on is a good start at controlling this, so it was doubled. That also allowed the dose of prednisolone I was on to be dropped quite a bit (I was on 75mg a day, which is a very high dose, for over a month and I had all the usual side effects – moon face, rapid weight gain particularly around the stomach, severe acne, mood swings…). It was also decided that I should do a four week course of Rituximab, which is a type of chemotherapy, but one that specifically destroys B-cells so you do not get side-effects people tend to think of with chemotherapy. I am also given a dose of IVIG monthly.

While this was going on, I was referred to a lung transplant doctor. With how my lungs are, there is not much room for further damage to be done before I would have significant difficulty doing every day tasks. There is only one case in the literature of someone having a lung transplant due to the follow-on effects of Castleman’s disease, so there is not much information to go on. The summary of my meetings with the lung transplant people was that I would not be a good candidate if my condition continued getting worse because there was nothing to say that my immune system would not just destroy the “new” set of lungs too. But if my condition stayed stable, I would have enough lung function to not warrant the risk of having a lung transplant now, particularly because living ten years with a lung transplant is a very good success. So ideally, the progress of my lung function decline due to the autoimmune condition would be halted now and I would only require a transplant later in life when the effects of the natural decline in lung function hits me earlier than most people because I will be starting from a worse base.

As my lungs no longer work very well and the sitting around in hospital and at home recovering did nothing to help my fitness, I was sent to a pulmonary rehabilitation program. This is a combination of cardio training and muscle building (as the fitter you are, the better you can handle low oxygen levels) and education (e.g. in how to manage shortness of breath, nutritional advise, some group counseling). Regaining fitness is particularly difficult when you have difficulty breathing on any moderate exercise. An interesting thing that I found out (although is really quite obvious when you think about it) is that the large muscle groups in your legs require a lot of oxygen, and they are used in everyday activities, so strengthening them can help you coping with shortness of breath. I also found out that a side-effect of being on high doses of prednisolone is muscle atrophy (wasting) so that would not have helped me in that regard.

And that about sums up where I am up to now. My lung function tests over the last six months have been quite stable so it looks like my current medication regiment is working. I have also noticed some improvement in my ability to do things like walk up a set of stairs without becoming short of breath, but I still have difficultly walking up hills or even walking fast on the flat. Hopefully keeping up with the exercise routine I am doing will see some more improvement, but there are really no guarantees there. My skin issues have mostly cleared up, leaving only a couple of small ulcerations on my tongue. The surface of my eyes also healed, but I am left with extremely dry eyes and have to continuously put in eye-drops. The only new issue I have had lately is an increased resting pulse rate (compensating for a lack of oxygen getting to my bloodstream). But multiple scans of my heart show there is no heart disease or any other issues so that is not too bad.

What can you do to help? There is not a lot that can be done for me personally, but I recommend the following:
1) Support medical research for rare diseases. Rare diseases are hard to diagnose, hard to treat and, as someone who does medical research, I know they are near impossible to get funding to do research into. If you see someone collecting for research into a rare disease and can spare a bit of money, give a donation.
2) Donate blood if you can. The monthly treatments of IVIG I receive comes from the plasma of a large number of donors. And like all blood products, there is a greater demand that what is donated, so help out if you can. Needles are not all that scary…
3) Sign up to donate your organs. The number of people needing organ donations far exceeds the number that donate. Your organs are not that much use to you once you are dead and with more people donating the organ matches will be better and the success rates higher. Also, tell your family that you want to donate your organs as with the current laws in most countries they will get the final say. I have always supported the idea of changing the law to make donation an “opt-out” process rather than the current “opt-in” and even removing the families right to override your decision.

And probably being more optimistic than anything…

4) Support stem cell research. I do not think the approach of “inject stem cells and hope for improvement” will work in my case but the growing of new organs from an individual’s own stem cells is a promising area of research and it would completely remove transplant rejections. Making a set of lungs would be very difficult, but recently a person had their trachea replaced with one built artificially from stem cells, so progress is being made.
5) Support artificial lung research. If I can not get a genuine pair of second-hand lungs in the future, then I will settle for being part machine. That would be cool!

The Great Pacman Bug Hunt of 2012

This is a story about a recent issue discovered in pacman, the Arch Linux package manager, and the difficulties we had hunting it down… The story is long, but so was the process of finding the bug.

It all started on a warm summer’s night (in my timezone and location… – it was probably cold and daytime for the other main pacman developers) with the reporting of FS#27805: “[pacman] seg faults when removing firefox”. Of course, my initial reaction was “bull shit” as we all know there are no bugs in the pacman code. But this was only a couple of weeks since pacman-4.0 was moved into the Arch Linux [core] repo so there was an ever so slight possibility it was real.

Luckily for us, the user reporting the bug was very helpful and installed a version of pacman with debugging symbols and gave us a full backtrace. It was very clear where the segfault was occuring:

#0 0xf7fbd4e7 in _alpm_pkg_cmp (p1=0x8128aa0, p2=0x0) at package.c:644

That function is called in the package removal process when we check that a file that is going to be removed with a package is not also owned by another package (which would require someone using -Sf when they should not). If the package in the local database is the same as the one being removed, we do not need to run this check, and hence the test. As you can see above, for some reason _alpm_pkg_cmp is being passed a null pointer as the package from the local database and KABOOM!

So the question was, how do we get a null value for the package from our local database? Given pacman runs through the list of local packages on each package removal, this null entry must have been generated on the removal of the previous package. Here is a bit of background on how package information is stored in pacman. Package information is stored in a hash table that also provides access to the data as a linked list. This provides us with fast look-up by a package’s name but also allows us to loop through the (generally sorted) package list. Now the hash table code is fairly new (first introduced in pacman-3.5) and the removal of items from a hash with collision resolution done by linear probing is not straight forward, so there could be a bug. Dan pointed his finger my way as I wrote the original hash table code and I pointed my finger his way as he made optimizations to the removal part. But it turns out that both of us were not thinking too hard. It is the list that is being corrupted and that has items removed using code that has been around for years. Despite that, the whole hash table and linked list removal code got an in depth review and no issues were found.

We were stumped. Looking at the the debug output from pacman, we could see that a file that actually did not exist on the system was being “removed” right before the crash, but that is not uncommon and appeared to be handled correctly so was unlikely to be the cause. So back to the reporter to see if we could get more information to replicate. He was very helpful and provided us with a copy of his local package database. We created a chroot with exactly the same packages and had no luck replicating. The user even provided us with a complete copy of his chroot where the error was occurring, but again there was no luck replicating. It must be something specific to that users system. Right? Well, even re-extracting the tarball of the chroot the user provided us onto his own system made the bug go away. All in all, a great candidate for being “not a bug”….

Until on another warm summers evening, while being my usual extremely helpful self on IRC, someone mentioned they were getting a segfault while removing packages. A bug report was filed and, again, the user was extremely helpful and the backtrace provided was exactly the same. A core dump showed us there was definitely something wrong with the linked list. Well… bugger! This bug appears real. Again the red-herring of the removal of a non-existent file was shown in the debug log, but it would be very, very strange for that to break the linked list of package information so was ruled out.

It was time to find a reproducer! So I created a chroot and set this script running:

ret=0
while (( ! ret )); do
  pkg=$(pacman -Sql extra | shuf -n1)
  pacman -S --noconfirm $pkg
  pacman -R --noconfirm -s $pkg
  ret=$?
done

Within five minutes I could replicate the segfault. (It turns out I was very lucky as I ran the same script again for over four hours and did not strike the issue.) Now it was time to get debugging!

The first thing I did was print some debugging info in the linked list node removal code, but for some reason the node removal just before the segfault did not print anything. I was only printing information when removing a node from the middle of the list (because that is where the package causing this issue was located), but just to be sure I also added debug statements for the case of removing the head and tail nodes. And then pacman told me it was removing a node from the end of the list… “Why do you think that package is a the end of the list pacman?”, I asked. “Because the head node’s prev entry tells me it IS the end of the list”, replied pacman. “Oh, crap”, I said. “So it does!” Something was clearly wrong here.

It was time to investigate all removal operations on that list. So I printed the entire linked list before and after each package removal and found the error actually occurred before the removal operation even started. The initial list of the local database passed to the removal operation was already broken with the pointer to the tail entry not pointing to the tail. That was good to know as we had thoroughly reviewed the removal code and not found any issues.

This lead me to believe that the error must occur when reading in the local database. Next step: print out the linked list at the end of reading in the local database. But that was completely fine. So somewhere between reading in the local database and using it, things got broken. And, what do we do with the local database between reading it in and removing items from it? The only place where we modify the local database between those points is when it gets sorted by the package names. Sure enough, the pointer to the tail of the linked list is good going into the sort and bad coming out.

This limited the error to two functions: alpm_list_msort or alpm_list_mmerge. These implement a merge sort. Essentially alpm_list_msort recursively calls itself, dividing the list up into smaller pieces until it can not be divided any further and they are then they are merged in sorted order by alpm_list_mmerge. I had just started staring at the code when I saw something that seemed too obvious for such a hard to track down bug. My exact words on IRC were “I think I can fix this…”. And sure enough I could.

It turns out that when alpm_list_msort split a list into two, it did not set the pointer to the tail nodes in the two new lists correctly (or at all…). So a two line addition and we have the bug fixed. It turns out this bug had been present since the start of 2007. So I am still slightly amazed that we did not see it before now and when it did appear that we got a second report of it so quickly.

And why could we not reproduce the issue even with a copy of a chroot where it was occurring? It is entirely dependent on the order the directory entries are returned from the disk. This determined which package was pointed to as the “tail” of the sorted package list. The package incorrectly referred to as the tail had to be removed during a removal operation, and also not be the last package removed, to expose the bug. Given most systems will have many hundreds of packages on them and removal operations tend to involve one or a few packages, this is a fairly rare occurrence. But even if it occurred only a fraction of a percent of removal operations, I think we should have ran into this bug before now. I guess more people probably did experience the issue, but then could not immediately replicate and did not experience the issue again so did not report it.

And that is the end of the story of one of the most frustrating bugs I have ever managed to track down. A big thank you to the two users who installed versions of pacman with debug symbols and provided us backtraces, coredumps and entire chroots! Without their help, we would probably still be not entirely convinced that the bug was real and it would still be hiding away in the pacman source code.

Disabling Junk Filtering With Hotmail

What many people do not realize is that my archlinux.org email address is really just an alias for a Hotmail account. That’s right… I do my Linux development with a Hotmail address! Deal with it… I am not changing.

Recently I got annoyed at the Webmail extension for Thunderbird as it breaks every time the Hotmail site makes a change or a new version of Thunderbird is released (so generally not the extension authors fault). The final straw was when attachments were only being partially downloaded, which I only noticed when some photos I was sent had black bars on them. Then I checked my WordPress backups that are automatically emailed to me and, sure enough, I could not open the zip file because it was corrupt.

Fortunately, your Hotmail emails can be retrieved by POP3 and this had become an option for me as my work no longer blocks that port. Unfortunately, that only downloads emails in your Inbox and not your Junk folder. That should be easy to work around… Just turn of Hotmail’s junk filtering. Right? Well, no… Hotmail has two junk email settings – “Standard” and “Exclusive” – and neither of these is equivalent to “Off”. The “Standard” filter manages to catch about half of the spam I receive and a bunch of valid emails from mailing lists I am subscribed to. I could log into Hotmail every so often and flag the valid emails as not junk (annoying), but Hotmail will delete anything in your Junk folder after ten days (really annoying…).

How to work around this crap? I found in Hotmail’s options there is a item to set up “Rules for sorting new messages”. It appears if an email matches one of these rules, the rule is enacted without running the spam filter. So the spam filter can be disabled by adding the following rule: Move messages to Inbox if sender’s address contains “@”.

That means I am now receiving all of my emails. So, if you send me an email and I do not respond, it is now definitely because I am ignoring you.

Xenoblade Chronicles – A Completionist’s Nightmare

Caution: Spoilers follow!

If you are looking for a good game and like JRPG’s, then you can not go past Xenoblade Chronicles. Well, you can if you live in North America where it still has not been released yet and will not be released until April… Now you know how it feels to live in Australia!

Now I have that off my chest, lets get back to the game. Xenoblade is a game of epic scope, both in terms of the shear size of the worlds that you can explore and in terms of the number of things there are to do. And there are many, many, many things to do… To get near “completing” the game, you will need to spend well over 100 hours, making the game quite good value for money.

I am not going to do a detailed review of the gameplay as those are already available elsewhere. Instead I will cover the various elements there are to collect in the game. This is probably an unhealthy obsession of mine that started with Pokemon (Gotta Catch ‘Em All…), but is often an aspect of games that I enjoy more than the primary game itself provided it is not too monotonous.

Party Upgrades: Each playable character in the game has multiple upgrades that can be found (excluding basic things like level and equipment). There are upgrades to each of the “Arts” that are used during battles, which are purchased using “AP” that you collect by defeating monsters or from other tasks in the game. Before the Arts can be upgraded to their full potential, you need to learn the Intermediate then Advanced levels through the use of Books. The Books for the Intermediate level can be purchased at various stores throughout the game, but the Advanced level books are only dropped by strong monsters. At least two are dropped at a low frequency by a monster that appears only once in the game, so unless you know that beforehand, the chance of completing this upgrade is negligible. I collected all Books but have not purchased the full upgrades yet as collecting the required amount of AP would become tedious.

Each character also has a set of “Skills” that are like innate abilities that improve your battle prowess. There are learnt by collecting “SP” when defeating monsters and filling up the “skill tree”. Each character starts with three skill trees, or sets of skills of a given type, but two additional skill trees can be earned for each character. Enough “SP” will be earned to fill up most of the characters skill trees just by playing the game, but a couple of characters would require repetitive monster killing to complete.

Quests: There are 480 quests given to you by various NPCs during the game. Some are essential to complete in order to progress through the game, but others are just useful for gaining experience/money/items. What is really annoying are that some quests only appear under certain conditions. I do not mind those that are mutually exclusive (i.e. you can complete one of two quests and it really does not matter which), but those quests that only appear if you do something in a particular way (with no real indication of what that is…) are… well… I can not find a polite word to describe them. Then there are timed quests. These become unavailable (without warning) once you reach certain points in the game. So if you are wanting to complete “all” quests, you need to do each quest as soon as you are assigned it and spend lots of time exploring each region to make sure you have talked to everyone. I believe I completed all possible quests for a single play-through apart from one that was unavailable as I made an “incorrect” choice during the game.

Affinity: There is an “affinity” system that essentially measure how much people like each other. Importantly from a gameplay perspective is how much the people of each region like you and how much the members of a party like each other. How much people of a particular region like you determines the available quests and items available for trading. This is improved by talking to the various NPC who have names and completing quests. The affinity between members of your party is improved by helping each other in battle and through the completion of quests together. It is not a super-important area of gameplay although it does let characters use skills known by other characters and allows you to see “Heart-to-Hearts” (see below).

Region Maps: There are around 20 areas (depending on how you count them) to explore during the game. Each of these regions had a number of “Landmarks” and “Locations” for you to find to unlock the complete map to the region. The Landmarks serve as warp-points, which avoids much mindless wandering from place-to-place. Almost all of these would be found during normal gameplay and the remaining few during completion of quests.

Collectopaedia: Each region has a list of items that can be collected throughout it. Collecting one of each of these fills in the Collectopaedia. Just like the Quests, there are points during the game where access to the areas becomes no longer possible (without warning…) so it is important to collect these as you go. There is also a selection of items needed to complete this that can only be traded for, with one requiring an item to trade that can only be found by defeating the strongest monsters in the game (and is the one item I have yet to collect).

Heart-to-Hearts: These are cut away scenes showing conversations between characters that are supposed to provide extra insight to their inner thoughts… You get to chose various answers that direct the outcome of these conversations, although I never actually read the text so I have no idea how much your choice mattered. What I did notice was that the “acting” during these interactions was horrible.

Unique Monsters: Now this is a fun part of the game! There are 157 “unique” monsters in the game. Some are truly unique in that they only appear once during a particular quest, but others consistently respawn. There are five of these monsters that have levels higher than your characters maximum level and it is these five I have left to beat (although I have not attempted them yet…).

Achievements: The game keeps track of your “achievements” as you progress through the game. These are separated into two types, Trials and Records. There are 50 Trials that basically cover working through all the collections above so if you are going to complete those then the Trials will get completed too. The 150 Records involve things like defeating a certain type of enemy a given number of times, using a given type attack a certain number of times, raising Skill and Arts to maximum levels and collecting crystals and crafting them into gems (an area of the game that is full of mystery as far as I am concerned…). Many of these fall into the tedious repetition category so I still have about 30 to complete.

That is a lot of stuff to do… As I said above, it will take substantially more than 100 hours if you want to do all of this. But I say it is definitely time well spent.

Panic

Tweet

How to make a website owner scared!

Posted in Tweet on by Allan Comments Off on Panic

Secure WordPress Administration For Free

Many months ago I noticed that I logged into my blog over plain HTTP and thought to myself that I really must do something about that one day. And that day is… well… a couple of days ago! I honestly was never really too concerned about logging in insecurely as the chances of anyone actually wanting to gain access to this blog and being in a position to exploit the insecure login is minimal. My guess would be that the majority of self-managed WordPress installs are administered over plain HTTP.

So apart from general apathy, what kept me from fixing this? Cost was probably the main issue… Any cost for a SSL certificate would not be particularly justified in my case. I also did not want to use a self-signed certificate as I find the security warnings that all web browsers give about untrusted certificates annoying enough to not want them on my site. That also rules out the free SSL certificates from CAcert, as the CAcert root certificate is not included by most browsers by default.

Then I saw a post somewhere about the free certificates given out by StartSSL. The price is right and the root certificate is commonly included so all seems good. There is not much actual validation that goes on to get one of these – my email and domain name were “verified” by sending emails… – so they would not be good for any site where trust is actually needed (such as anything where any personal and financial data are being collected).

Once validated, all I had to do was provide a CSR and they provided me the certificate. My webhost then uploaded that certificate and broke everything! The HTTPS version of my site was giving the error “ssl_error_rx_record_too_long”, which is actually quite uninformative as it covers a wide range of actual issues, and the HTTP version for some reason lost all access to files even thought they were clearly still there when I checked. This took me a few hours to notice as I had to wait for the DNS entries to propagate, so the issue was reported at 5pm on Friday the 30th of December… I really thought my website would be down until the 3nd of January when the support desk reopened, but everything was fixed a few hours later. So good service given what I pay, but the whole issue could have been avoided with a simple check at their end once the SSL certificate was installed.

Once you have your SSL certificate installed and ready to go, making WordPress enforce SSL usage for all administration tasks is simple. Simply add the following to your wp-config.php file:

define('FORCE_SSL_ADMIN', true);

Now all your blog administration is secure(ish). The final thing to do was to check whether browsing my website using HTTPS worked… No, it did not! I was getting messages about the site only being partially encrypted. A quick search showed I serve all my images using the full URL rather than a relative one. I did this because a certain Linux distribution’s Planet feed did not show images otherwise (or at least that was the case a long time ago – I have not tested lately). I could go through and adjust all my image links to use HTTPS, or just disable HTTPS access to my website. I chose the latter as nothing on my site is that important that it needs to be encrypted and I thought it would be the quicker option… Several hours later and this is the rule you need to add to your .htaccess file to achieve this:

RewriteCond %{ENV:HTTPS} on [NC]
RewriteRule !^wp-(admin/|login.php|includes/|content/)(.*)$ http://allanmcrae.com%{REQUEST_URI} [R,L]

The only real trick there is that the WordPress login and administration interface uses files from the wp-includes and wp-contents directories so they need to be excluded from the RewriteRule.

So… remember how I said self-signed certificates were annoying as all visitors to the site would get a warning. Well, now I force HTTP usage, that whole argument is irrelevant as only I would see the SSL certificate when I access the administration interface. But I at least have the option of serving parts of the site over HTTPS using a recognized certificate if I ever feel the need.