Author Archives: Allan

Basic Overview of Pacman Code

I am far from knowledgeable about most areas of the pacman code-base, so whenever I want to implement a new feature I first have to sit down and walk step-by-step through a transaction and figure out the code path. Given I have now done this several times, I thought it a good idea to post it for further reference.

Here is my brief overview of what happens with a pacman -S <pkg> transaction:

pacman/pacman.c:
 -> int main(int argc, char *argv[])
      - parse command line
      - parse config file
 
pacman/sync.c:
 -> int pacman_sync(alpm_list_t *targets)
      - check we have a pacman database
      - check target list for SyncFirst packages
 
 -> static int sync_trans(alpm_list_t *targets)
      - initialise transation
      - process the target list
 
libalpm/trans.c:
 -> int SYMEXPORT alpm_trans_prepare(alpm_list_t **data)
      - perfoms checks (via _alpm_trans_prepare)
         - requested packages have valid architecture
         - all deps are available
         - deps do not directly conflict (not files)
 
 -> int SYMEXPORT alpm_trans_commit(alpm_list_t **data)
 
libalpm/sync.c:
 -> int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data)
      - download needed files
      - deal with deltas (if any)
      - check package integrity
      - check file conflicts
      - remove packages (conflicts/replacements)
      - install targets

Doing a pacman -Syu takes you down exactly the same code path with the only difference being that the package databases are updated and the list of packages to be updated is calculated in the pacman_sync function.

While that is a fairly basic overview, it has more than enough detail for me to locate where I should implement checking for free disk space before proceeding with a package install (which has been a long time feature request for pacman).

I hope this also shows people that the pacman code is not that complex. There are quite a few old bug reports/feature requests in the bug tracker that are obviously very low on the developers priority list and are good candidates for new contributors. Just step through the code until you find the relevant section and then get started!

Will McDonald’s Back Down?

Ever been to McDonald’s right after they stopped serving the breakfast menu and been really disappointed at their lack of flexibility in menu change timings? Nor have I… but I am told the conversation goes something like this:

   ”No sir, we can not give a hashbrown as we stopped serving breakfast about 15 seconds ago.”
   ”What about that leftover hashbrown sitting right there?”
   ”We stop serving breakfast at 10:30am. That hashbrown is dead to us.”

But do they make any exceptions? I think we may soon find out. Someone is in the process of threatening to blow themselves up in Brisbane, forcing a closure of part of the CBD including some ferry terminals. Apparently he has made a request for a Bacon and Egg McMuffin, which is a good choice as they have less fat content than the similar Sausage and Egg McMuffin. That and bacon is awesome. However, despite this request being made after the 10:30am cut-off for McDonald’s breakfast, the police are still trying to fulfill it.

Things just got interesting… If McDonald’s folds and provides the McMuffin then we have a baseline for how much pressure must be exerted before they serve breakfast after 10:30am. Then we just need to bisect the pressure scale to determine the “caving threshold” above which we can obtain post 10:30am McDonald’s breakfast. If McDonald’s does not provide the McMuffin, I suggest we do not escalate this further and just call an end to this highly scientific experiment.

Big Python Transition In Arch Linux

After a long time of planning and a massive rebuild, the “Great Python Rebuild of 2010″ has finally made its way into the [testing] repo for Arch Linux. And I do mean a massive rebuild… weighing in at 587 (and still growing) packages, that is a little more that 10% of our repos! That is bigger than the combined libjpeg/libpng rebuild earlier this year.

So what exactly has happened? Two things. Firstly, the python package now is from the python-3.x branch (currently 3.1.2). That means /usr/bin/python will point to /usr/bin/python3 being the default python-3.x interpreter. A new python2 package has been introduced containing the latest from the python-2.x branch (currently 2.7.0), with its primary binary being /usr/bin/python2. Note that also is an upgrade from python-2.6.5 which is currently in [extra]. While we were rebuilding everything anyway, I also enabled UCS4 (unicode) support in python2.

WHAT!!! Am I crazy? Python-3.x as the default /usr/bin/python? Well, my sanity may be questionable but the principles behind this transition are sound. Arch Linux is unashamedly bleeding edge so we do big transitions like this first. Other distros can follow in our wake. Also, python-2.7 will be the last in the python-2.x release series so we are following upstream intent and moving the focus to the python-3.x series.

Doing this transition now also allows us to gradually rename our packages. e.g. python-foo is still a package for python2 and will eventually need renamed to python2-foo. That would be much more difficult if we waited until we had a heap of (e.g.) python3-foo packages in the repos. Not being able to make a clean break in package naming like that is one of the disadvantages of a rolling release distro.

For all packages in the official repos, this transition should be very smooth. Packages from the AUR may need to be rebuilt to point at the /usr/bin/python2 binary rather than /usr/bin/python. A fair portion of packages will recognise the binary is named python2 and make all the required adjustments. Others will recognise the environmental variable PYTHON=python2 and use that. Finally, some packages will need a sed line to adjust scripts like:

sed -i -e "s|#![ ]*/usr/bin/python$|#!/usr/bin/python2|"
       -e "s|#![ ]*/usr/bin/env python$|#!/usr/bin/env python2|"
  $(find $pkgdir -name '*.py')

Of course, your own python scripts will need to point at /usr/bin/python2. However, by doing this you may run into portability issues across distros. There does not appear to be an easy solution for that at the moment. It seems that while most (all?) distributions include a /usr/bin/python3 link to their python3.x binary, none do the same thing for python2.x. Either create your own symlink in your path for those distros or even better file a bug with them asking for such a symlink. They are going to need one in the future…

If you are using the [testing] and [community-testing] repos go ahead and upgrade to the future of python. If see any of the following in one of our supported packages, report a bug:

  • files linked to libpython2.6.so.1.0
  • files with a “#!/usr/bin/python” or “#!/usr/bin/env python” shebang
  • files in /usr/lib/python2.6 or /usr/include/python-2.6
  • depends/makedepends/optdepends on “python” instead of “python2″
  • packages that no longer work…

So far it looks like the rebuild has gone remarkably smoothly. A big thank you to all the developers and trusted users who helped out with the rebuilds. It also shows the power of our new repo setup on the mirrors. Now moving all those packages between repos just moves some symlinks so syncing to mirrors is very fast. So your favourite mirror should sync very quickly unlike with big rebuilds in the past.

But there is no rest for the wicked… We will be onto rebuilds for db and heimdal next!

“Finished” Super Mario Galaxy 2

Caution: Spoilers follow!

I know my limits so have declared 241/242 stars in Super Mario Galaxy 2 as being finished… The final level is only for obsessed people with too much time on their hands!

I am still not sure how this game was rated as highly as it was (although I do think it was a great game). It has a 97% average on Metacritic, making it amongst the highest rated games of all time. However, the only real addition to the gameplay from Super Mario Galaxy is “hide-and-seek”. I played that as a kid years ago so that is hardly novel… And it is not just one type of hide-and-seek; it is used multiple ways. First you have to find the Comet Coins to unlock a challenge on each world. Then once you have done all those levels (and beaten the final boss yet again – probably for the third time…), there becomes another 120 green stars that are hidden across the game. You think with all that practice the game designers would be good at hide-and-seek, but a fair portion of the things you have to find are clearly visible in the opening sequence for each world, which kind of ruins the whole point. In fact, I’m not sure there was a point at all other than to force you to constantly repeat the same levels.

But the game was saved by a review I watched beforehand, where we are told to think of “cake” as a Mushroom Kingdom codeword for “sex”. It made the Bowser’s final words rather amusing!

cake

Posted in Games on by Allan – Comments Off on “Finished” Super Mario Galaxy 2

Recovering Files From an NTFS Drive

Important lessons for computers owners:

  1. Set up a backup system
  2. Use that backup system regularly

It turns out that while I set up a backup system for my wife’s computer, the last time it was ran was sometime late 2009… So when the primary Windows partition on that computer no longer booted; was not recognised as a Windows install by the Windows install CD rescue option; and would not mount using the ntfs-3g driver from Linux, I knew I was in trouble! Luckily, I had split the Windows install to have a separate partition where most of her data (photos, music, documents) were sitting and that partition was still mountable. So, in the worst case scenario, the total damage would be limited to Firefox and Thunderbird profiles and whatever was saved on the desktop.

Recovering more data from the hard-drive was not particularly difficult as the hard-drive still basically worked. In that case, the first step in any data recovery attempt is to create a disk image of the broken partition. This prevents any further damage to the hard-drive making you life worse. A good tool for the job is GNU ddrescure (which may be named ddrescue or gddrescue depending on your Linux distribution). Make sure you have the GNU version as the other software with the same name is less user friendly. It can take a long time to take an image of a hard-drive so having some nice output and being able to stop and restart at any stage are quite essential. Creating an image is as simple as running (as root):

ddrescue -r3 /dev/sda2 imagefile logfile
The log file shows that the hard-drive had quite a few corrupt parts, all relatively small and close together so my guess is some sort of physical damage preventing those being read.

# Rescue Logfile. Created by GNU ddrescue version 1.13
# Command line: ddrescue -r3 /dev/sda2 imagefile logfile
# current_pos current_status
0x30D31E00 +
# pos size status
0x00000000 0x309EB000 +
0x309EB000 0x00001000 -
0x309EC000 0x00073000 +
0x30A5F000 0x00002000 -
0x30A61000 0x00073000 +
0x30AD4000 0x00002000 -
0x30AD6000 0x00073000 +
0x30B49000 0x00002000 -
0x30B4B000 0x000FC000 +
0x30C47000 0x00001000 -
0x30C48000 0x00074000 +
0x30CBC000 0x00001000 -
0x30CBD000 0x00074000 +
0x30D31000 0x00001000 -
0x30D32000 0x7F5409E00 +

So now you have an image of your hard-drive, it is time to get those files out. Given the disk image was relatively complete, I went for software called Sleuth Kit. It has a frontend called Autopsy, which I found fairly useless apart from browsing the data. To use it, start Autopsy by pointing it at directory where all its files are to be stored and then point your web browser at the relevant place:

mkdir autopsydir
autopsy -d autopsydir
firefox http://localhost:9999/autopsy

While browsing the data was good to confirm that most of the files I wanted were still there, I just wanted to extract every single file possible from the image and I would then copy all the relevant stuff over to the new computer as necessary. Slueth Kit has some command-line tools for doing that; fls for listing files and icat for getting them. Using these, you can extract the files one at a time… but a simple script will automate extracting everything.

#!/bin/sh
IMAGE=../imagefile
fls -urp $IMAGE |
while read type inode name; do
   echo $name
   case $type in
      d/d) mkdir -p "$name" ;;
      r/r) [ ! -f "$name" ] && icat $IMAGE $(echo $inode | sed 's/://g') > "$name" ;;
   esac
done

That will take a long time to run, especially if it is your primary Windows install drive as it will have lots of small files to extract. So I made that script so that it can continue where it left off. Just do not stop one run in the middle of an important file or it will not finish extracting it. You might want to also touch hiberfil.sys and pagefile.sys in the extraction directory first as they are relatively useless and will take up a few gigabytes each.

So everything appears to have been recovered and I survived the lack of backup! Windows 7 has a good feature that reminds you to set up a backup and run it regularly, so hopefully I will not need to do this again.

Posted in Software on by Allan – Comments Off on Recovering Files From an NTFS Drive

Australian Election

This is a good summary of the build-up to the Australian election:

The world needs more animated news. For people unfamiliar with Australian politics, there is barely any exaggeration in that clip. Also, the Sex Party was actually a reasonably good choice if you look at their policies.

Not that I get to vote in any of this… I am proudly not an Australian citizen!

Posted in General Rant on by Allan – Comments Off on Australian Election

Undefeatable New Zealand!

Guess which team was the only one to be undefeated at the “Soccer” World Cup? That’s right… New Zealand!

(We did not win a game either but that is not the point.)

Nintendo Australia – You Suck!

Lets look at the release dates of Super Mario Galaxy 2 around the globe (according to Wikipedia)…

  • North America: May 23, 2010
  • Japan: May 27, 2010
  • EU: June 11, 2010
  • Australia: July 1, 2010

But Australia is in the same Wii game region as the EU… the release is exactly the same. So what the hell Nintendo! You are lucky that the game will be awesome so I can not protest.

The delay from the North American release data is something I have learned to live with, as EU region releases are always delayed (supposedly due to having to do the translations). But at least when the Smash Bros Brawl release date in Australia was four months behind the North American release, the EU suffered along with us.

Posted in Games on by Allan – Comments Off on Nintendo Australia – You Suck!

Pacman 3.4.0 Released

As Dan has already posted about, pacman-3.4.0 has been released. There are a bunch of new features that I am really enjoying.

Firstly, when updating it database, pacman will only extract the new entries. This is similar to what Xyne’s rebase script does (without all the extra output). I had not realised how awesome this feature was until I updated my chroots this morning. It speeds the process up immensely. The chroots using pacman-3.4 extracted the [extra] repo database with a barely noticeable pause while those using pacman-3.3 took a while.

The other feature that I am enjoying is the addition of a functional ‘which’ to the file ownership query. In the past, to find the owner of a binary in my path I would do something like pacman -Qo $(which makepkg) or provide the full path manually. Now pacman will search for binaries in your path automatically, so this is achieved with pacman -Qo makepkg.

Installing packages with pacman -U has received a nice overhaul, allowing pacman to handle package replacements and install needed dependencies all in one transaction. No more removing a package with pacman -Rd and then installing its replacement.

And makepkg also received its share of upgrades. It now automatically exits on build/packaging errors in PKGBUILDs so there is no more need to have “|| return 1” after the commands. Package splitting has improved with pkgver, pkgrel and arch now being able to be overridden and being able to only build subsets of a split package.

Of course, many other features made it into this pacman release. As always, many changes will hopefully never be noticed by a user (e.g. checking a package architecture matches the system architecture before installing, a major rewrite of the pacman bash completion, overhaul of tests in makepkg, more configurable library stripping during packaging), but all these are very useful contributions. See here for a more detailed summary of the changes and the git log for all the details of changes.

A pacman-3.4.0 package is currently in the [testing] repository for Arch Linux. We all know pacman releases are bug free (as the two patches already in the 3.4.1 queue can attest), so look forward to it being in a [core] repo near you in the not too distant future.

How Do You Check E-mails?

Saw an infomercial for Australia’s biggest ISP (Telstra BigPond) today and learnt something amazing. The “expert” talking to the presenter said something like:

People do a lot of things on-line these days. Watching movies, checking emails, browsing…

I was a bit taken back… People use the internet for checking emails? I have been doing it wrong all these years!